Rule-based deduplication of article records from bibliographic databases
Open accessThe DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
Abstract
We recently designed and deployed a metasearch engine, Metta, that sends queries and retrieves search results from five leading biomedical databases: PubMed, EMBASE, CINAHL, PsycINFO and the Cochrane Central Register of Controlled Trials. Because many articles are indexed in more than one of these databases, it is desirable to deduplicate the retrieved article records. This is not a trivial problem because data fields contain a lot of missing and erroneous entries, and because certain types of information are recorded differently (and inconsistently) in the different databases. The present report describes our rule-based method for deduplicating article records across databases and includes an open-source script module that can be deployed freely. Metta was designed to satisfy the particular needs of people who are writing systematic reviews in evidence-based medicine. These users want the highest possible recall in retrieval, so it is important to err on the side of not deduplicating any records that refer to distinct articles, and it is important to perform deduplication online in real time. Our deduplication module is designed with these constraints in mind. Articles that share the same publication year are compared sequentially on parameters including PubMed ID number, digital object identifier, journal name, article title and author list, using text approximation techniques. In a review of Metta searches carried out by public users, we found that the deduplication module was more effective at identifying duplicates than EndNote without making any erroneous assignments.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“In a recent study by Jiang [ 23 ], the authors also found that EndNote, for the same reason, had erroneously assigned unique records as duplicates.”
From Better duplicate detection for systematic reviewers: evaluation of Systematic Review Assistant-Deduplication Module (2015) · cites this paper as [23]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adhi Harmoko, S., M.Komp, Jacquard, J. M., Zuse, K., & Eniac (2005) · Cited by 18,035
Grossman, P. (2009) · Discrete Mathematics for Computing · doi:10.1007/978-0-230-37405-8_1 · Cited by 8,834
Fellegi, I. P., & Sunter, A. B. (1969) · Journal of the American Statistical Association · doi:10.1080/01621459.1969.10501049 · Cited by 2,475
Duplicate Record Detection: A Survey
Elmagarmid, A. K., Ipeirotis, P. G., & Verykios, V. S. (2007) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2007.250581 · Cited by 1,267
Efficient clustering of high-dimensional data sets with application to reference matching
McCallum, A., Nigam, K., & Ungar, L. H. (2000) · Proceedings of the sixth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/347090.347123 · Cited by 1,096
Adaptive duplicate detection using learnable string similarity measures
Bilenko, M., & Mooney, R. J. (2003) · Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/956750.956759 · Cited by 926
Real-world Data is Dirty: Data Cleansing and The Merge/Purge Problem
Hernández, M. A., & Stolfo, S. J. (1998) · Data Mining and Knowledge Discovery · doi:10.1023/a:1009761603038 · Cited by 812
Newcombe, H. B., Kennedy, J. M., Axford, S. J., & James, A. P. (1959) · Science · doi:10.1126/science.130.3381.954 · Cited by 770
Interactive deduplication using active learning
Sarawagi, S., & Bhamidipaty, A. (2002) · Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/775047.775087 · Cited by 692
A SIMPLE ALGORITHM FOR IDENTIFYING ABBREVIATION DEFINITIONS IN BIOMEDICAL TEXT
SCHWARTZ, A. S., & HEARST, M. A. (2002) · Biocomputing 2003 · doi:10.1142/9789812776303_0042 · Cited by 508
Research works printed but not linked: 1
- Bhattacharya, 2004, Deduplication and group detection using links
Cited by
De-duplication of database search results for systematic reviews in EndNote
Bramer, W. M., Giustini, D., de Jonge, G. B., Holland, L., & Bekhuis, T. (2016) · Journal of the Medical Library Association : JMLA · doi:10.3163/1536-5050.104.3.014 · Cited by 2,103
“In the authors’ opinion, these methods are either very time consuming [5] or impractical, as they require uploading large files to an online platform [6, 7].”
Systematic review automation technologies
Tsafnat, G., Glasziou, P., Choong, M. K., Dunn, A., Galgani, F., & Coiera, E. (2014) · Systematic Reviews · doi:10.1186/2046-4053-3-74 · Cited by 472
“Heuristic [84], machine learning (ML) algorithms and probabilistic string matching are used to compare author names, journal, conference or book names (known as venue) and titles of citations.”
MethodMcKeown, S., & Mir, Z. M. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01583-y · Cited by 251
Rathbone, J., Carter, M., Hoffmann, T., & Glasziou, P. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-6 · Cited by 206
“In a recent study by Jiang [23], the authors also found that EndNote, for the same reason, had erroneously assigned unique records as duplicates.”
BackgroundIdentifying and removing duplicate records from systematic review searches
Kwon, Y., Lemieux, M., McTavish, J., & Wathen, N. (2015) · Journal of the Medical Library Association : JMLA · doi:10.3163/1536-5050.103.4.004 · Cited by 178
“, we believe false positives are more detrimental than false negatives because systematic reviewers want to maintain the highest possible recall in retrieval [7].”
BackgroundThielen, F., Van Mastrigt, G., Burgers, L., Bramer, W., Majoie, H., Evers, S., & Kleijnen, J. (2016) · Expert Review of Pharmacoeconomics & Outcomes Research · doi:10.1080/14737167.2016.1246962 · Cited by 132
(2023) · doi:10.1186/s12915-023-01686-z · Cited by 103
“SRA-DM has been shown to identify substantially more duplicates than Endnote (13, 16).”
BackgroundInfluentialCohen, A. M., Smalheiser, N. R., McDonagh, M. S., Yu, C., Adams, C. E., Davis, J. M., & Yu, P. S. (2015) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocu025 · Cited by 78
Self-citationsrBERT: automatic article classification model for systematic review using BERT
Aum, S., & Choe, S. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01763-w · Cited by 48
Smalheiser, N. R., Lin, C., Jia, L., Jiang, Y., Cohen, A. M., Yu, C., Davis, J. M., Adams, C. E., McDonagh, M. S., & Meng, W. (2014) · Health Information Science and Systems · doi:10.1186/2047-2501-2-1 · Cited by 47
“about the algorithms and code for de-duplication in Metta are described in a separate publication [17].”
MethodSelf-citationHair, K., Bahor, Z., Macleod, M., Liao, J., & Sena, E. S. (2021) · doi:10.1101/2021.05.04.442412 · Cited by 103
Searching for and selecting studies
Spijker, R., Dinnes, J., Glanville, J., & Eisinga, A. (2023) · Wiley eBooks · doi:10.1002/9781119756194.ch6 · Cited by 34
Evidence-based literature review: De-duplication a cornerstone for quality
Hammer, B., Virgili, E., & Bilotta, F. (2023) · World Journal of Methodology · doi:10.5662/wjm.v13.i5.390 · Cited by 31
Validity assessment of a computational system in the identification of duplicate studies
Escaldelai, F. M. D., Escaldelai, L., & Bergamaschi, D. P. (2023) · Escola Anna Nery · doi:10.1590/2177-9465-ean-2022-0143en · Cited by 21
Normalization of Duplicate Records from Multiple Sources
Dong, Y., Dragut, E. C., & Meng, W. (2018) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2018.2844176 · Cited by 24
“The record matching problem is also referred to as duplicate record detection [9], record linkage [10], object identification [11], entity resolution [12], or deduplication [13] and the truth discovery problem is also called as truth finding [14] or fact finding [15]—a key problem in data fusion [16], [17].”
BackgroundSelf-citationLiu, K., & El-Gohary, N. (2022) · Advanced Engineering Informatics · doi:10.1016/j.aei.2021.101496 · Cited by 18
Rodríguez, E. G., Ponjuán, D. T., Lahera, Y. M., & Jorge, R. A. (2019) · Scientometrics · doi:10.1007/s11192-019-03233-6 · Cited by 15
“The database deduplication strategy followed is consistent with methodologies in place for bibliographic record disambiguation (Jiang et al. 2014).”
MethodAvaliação de validade de um sistema computacional na identificação de estudos duplicados
Escaldelai, F. M. D., Escaldelai, L., & Bergamaschi, D. P. (2023) · Escola Anna Nery · doi:10.1590/2177-9465-ean-2022-0143pt · Cited by 13
Do Authors Deposit on Time? Tracking Open Access Policy Compliance
Herrmannová, D., Pontika, N., & Knoth, P. (2019) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1109/jcdl.2019.00037 · Cited by 13
“For example, in addition to the metadata we already use for the matching, we could utilise all other metadata available to us, such as abstracts, and employ looser matching techniques such as those used in article deduplication [9].”
MethodDeduplication of Scholarly Documents using Locality Sensitive Hashing and Word Embeddings
Gyawali, B., Anastasiou, L., & Knoth, P. (2020) · Open Research Online (The Open University) · Cited by 40
Proposed threshold-based and rule-based approaches to detecting duplicates in bibliographic database
Amin, M., Stiawan, D., Ermatita, E., & Budiarto, R. (2024) · Bulletin of Electrical Engineering and Informatics · doi:10.11591/eei.v13i3.7665 · Cited by 7
“The rule used in duplication detection is a rule for filtering a set of records [26], not a rule for performing the reasoning or inferencing process.”
BackgroundMethodAn automated data collection process for constructing graph data relying on LLMs
Ho, N. N. T., Nguyen, H., Ngueyen, N. M. C., & Nguyen, P. C. (2024) · CTU Journal of Innovation and Sustainable Development · doi:10.22144/ctujoisd.2024.323 · Cited by 3
“Some existing mechanisms are: exact matching (Ma et al., 2017), fuzzy matching (Chaudhuri, 2003; Ranjith, 2022), rule-based deduplication (Jiang et al., 2014) and Machine learning-driven deduplication (Akbar et al., 2023).”
Garcia, K. K. S., Henriques, K. M. D. D., Balieiro, A. A. D. S., de Pina-Costa, A., & Siqueira, A. M. (2024) · Malaria Journal · doi:10.1186/s12936-024-05133-4 · Cited by 3
Towards Structuring of Electronic Marketplaces Contents: Items Normalization Technology.
Cherednichenko, O., Yanholenko, O., Vovk, M., & Sharonova, N. (2020) · eNTUKhPIIR Repository (Kharkiv Polytechnic Institute) · Cited by 3
Duct-Taping Databases, or How to Use Fragmentary Online Data for Researching “Japanese” Videogames
Mühleder, P., Roth, M., Arndt, T., & Rämisch, F. (2020) · Journal of the Japanese Association for Digital Humanities · doi:10.17928/jjadh.5.1_61 · Cited by 2
“We used the categorization schema put forward by Yu Jiang et al. (2014) If a field has low accuracy, it is not a good idea to use it as a duplication indicator.””
BackgroundMethodUnderstanding human disease knowledge through text mining
Rodriguez‐Esteban, R. (2016) · Cambridge University Press eBooks · doi:10.1017/cbo9780511989421.004 · Cited by 4
Miary podobieństw łańcuchów znakowych a deduplikacja rekordów w bibliograficznych bazach danych
Kamińska, A. M. (2017) · Przegląd Biblioteczny · doi:10.36702/pb.463 · Cited by 2
Shobha, K., & Nickolas, S. (2019) · Smart innovation, systems and technologies · doi:10.1007/978-981-13-9282-5_61 · Cited by 2
Chung, N. H. T., Hieu, V. D., & Dung, N. M. (2019) · VNU Journal of Science Policy and Management Studies · doi:10.25073/2588-1116/vnupam.4201 · Cited by 2
Bertelmann, R., Boltze, J., Ceynowa, K., Christof, J., Faensen, K., Groß, M., Hoffmann, C., Koch, T., Kuberek, M., Lohrum, S., et al. (2021) · Publication Database GFZ (GFZ German Research Centre for Geosciences) · doi:10.12752/8150 · Cited by 2
Shen, J., Ling, F., & Hartmann, E. M. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.09.29.510210 · Cited by 2
“, “α” to “alpha”, “β” to “beta”) and (2) transliterate accented characters to ASCII equivalents (e.”
BackgroundMethodInfluentialO desafio da deduplicação de publicações criação e avaliação de um benchmark
Mena‐Chalco, J. P., Dias, T. M. R., do Canto, F. L., Moreira, T. H. J., & Segundo, W. L. R. D. C. (2024) · Encontro Brasileiro de Bibliometria e Cientometria (EBBC) · doi:10.22477/ix.ebbc.411 · Cited by 1
Barreto, C. A. D. S., Júnior, A. T. F., Melo, L. D. G., Santos, M. D. R., Carvalho, G. R., Filho, I. D. M. B., Neto, J. X. D. O., Malaquias, R. S., Gurgel, A. M., & de Lima, J. M. M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-96997-3_23 · Cited by 1
deduplicate.it: A Simple Open-Source Tool to Remove Duplicates From Literature Searches
Purkarthofer, D., Labenbacher, S., Bornemann‐Cimenti, H., & Landoni, G. (2026) · Campbell Systematic Reviews · doi:10.1177/18911803261484937 · Cited by 1
“Deduplication is a non-trivial issue, as meta-data is not consistently reported between databases with differences found amongst author names, journal abbreviations or page numbers (Jiang et al., 2014).”
Kirubakaran, A., & Murugaiyan, A. (2018) · Communications in computer and information science · doi:10.1007/978-981-10-8603-8_8 · Cited by 1
Short Review on srBERT: Automatic Article Classification Model for Systematic Review Using BERT
Choe, S., Aum, S., & Kim, J. H. (2022) · Asian Journal of Complementary and Alternative Medicine · doi:10.53043/2347-3894.acam90025 · Cited by 2
Science and art of writing the title of systematic review and meta-analysis
Tantry, T. P., & Garg, R. K. (2025) · Indian Journal of Anaesthesia · doi:10.4103/ija.ija_415_25 · Cited by 1
“[10] Example: “Probiotics for Irritable Bowel Syndrome in elderly: A Meta-Analysis.””
Citation Enrichment Improves Deduplication of Primary Evidence
Choong, M. K., Thorning, S., & Tsafnat, G. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-25660-3_20
Evaluation of unique identifiers used for citation linking
Madsen, H. H., Madsen, D., & Gauffriau, M. (2016) · F1000Research · doi:10.12688/f1000research.8913.1
Madsen, H. H., Madsen, D., & Gauffriau, M. (2016) · F1000Research · doi:10.12688/f1000research.8913.2
“Two studies analyzed deduplication via UIDs (DOI, PMID, and arXiv ID) and other match key elements in Mendeley (Hammerton et al. , 2012) and for a metasearch engine which covered five biomedical datadases (Jiang et al. , 2014).”
BackgroundEscaldelai, F. M. D. (2022) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/t.6.2022.tde-04042024-101557
Additional file 1 of srBERT: automatic article classification model for systematic review using BERT
Aum, S., & Choe, S. (2021) · Figshare · doi:10.6084/m9.figshare.16909956
Migration et enrichissement sémantique d’entités culturelles
Decourselle, J. (2018) · theses.fr (ABES) · doi:10.70675/8cafa02czf2a5z40f7z8b1dz23725ba5f726
Design in modular building components: a systematic review
Xing, D., Chang-Richards, A. Y., Lu, Y., & Zou, Y. (2026) · Construction Innovation · doi:10.1108/ci-06-2025-0257
An LLM-assisted framework for synthesis of sustainable development research on the Loess Plateau
Xue, W., Zhao, N., Wang, Y., & Xu, M. (2026) · Geography and sustainability · doi:10.1016/j.geosus.2026.100569
Metadata
- Authors
- Jiang, Y., Lin, C., Meng, W., Yu, C., Cohen, A. M., & Smalheiser, N. R.
- Affiliations
- Binghamton University (US)
- Corresponding author
- Smalheiser, Neil R.
- Venue
- Database
- Publisher
- Oxford University Press (OUP)
- Published
- 2014-01-01
- Volume
- 2014
- Pages
- bat086-bat086
- ISSN
- 1758-0463 (ISSN-L)
- PMC
- PMC3893659
- OpenAlex
- W2105224613
- Cited by
- 41
- References
- 20
- Citations by year
- 2026: 2 · 2025: 2 · 2024: 3 · 2023: 5 · 2022: 3 · 2021: 3 · 2020: 3 · 2019: 4 · 2018: 3 · 2017: 1 · 2016: 6 · 2015: 4 · 2014: 1
- FWCI
- 5.05
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Database (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- The Open Repository - Binghamton (Binghamton University) (submittedVersion)
- Funders
- National Institutes of Health
- U.S. National Library of Medicine (R01LM010817)
- Topics
- Physical Sciences › Computer Science › Information Systems › Data Quality and Management
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Library Science and Information Systems
- Keywords
- data deduplication, bibliographic records, rule-based matching, metasearch engines, metta, biomedical databases, systematic reviews, PubMed, Embase, CINAHL, PsycINFO
- MeSH terms
- Algorithms, Publications, Records, Databases, Bibliographic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.