The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index; 10 of the 11 sources its claims cite were checked for support.
Reducing systematic review burden using Deduklick: a novel, automated, reliable, and explainable deduplication algorithm to foster medical research
Open accessAbstract
BACKGROUND: Identifying and removing reference duplicates when conducting systematic reviews (SRs) remain a major, time-consuming issue for authors who manually check for duplicates using built-in features in citation managers. To address issues related to manual deduplication, we developed an automated, efficient, and rapid artificial intelligence-based algorithm named Deduklick. Deduklick combines natural language processing algorithms with a set of rules created by expert information specialists. METHODS: Deduklick's deduplication uses a multistep algorithm of data normalization, calculates a similarity score, and identifies unique and duplicate references based on metadata fields, such as title, authors, journal, DOI, year, issue, volume, and page number range. We measured and compared Deduklick's capacity to accurately detect duplicates with the information specialists' standard, manual duplicate removal process using EndNote on eight existing heterogeneous datasets. Using a sensitivity analysis, we manually cross-compared the efficiency and noise of both methods. DISCUSSION: Deduklick achieved average recall of 99.51%, average precision of 100.00%, and average F1 score of 99.75%. In contrast, the manual deduplication process achieved average recall of 88.65%, average precision of 99.95%, and average F1 score of 91.98%. Deduklick achieved equal to higher expert-level performance on duplicate removal. It also preserved high metadata quality and drastically reduced time spent on analysis. Deduklick represents an efficient, transparent, ergonomic, and time-saving solution for identifying and removing duplicates in SRs searches. Deduklick could therefore simplify SRs production and represent important advantages for scientists, including saving time, increasing accuracy, reducing costs, and contributing to quality SRs.
Overview
CoChat Summary
TL;DR
Deduklick is an NLP-based automated deduplication algorithm for systematic review references, benchmarked against standardized manual EndNote deduplication on eight heterogeneous datasets. It matched or exceeded manual performance while reducing average execution time from 70 minutes to under one minute.
Key findings
- Across eight datasets, Deduklick averaged 99.51% recall, 100.00% precision, and 99.75% F1; manual deduplication averaged 88.65% recall, 99.95% precision, and 91.98% F1.
- Deduklick’s F1 score was higher than manual deduplication in six of eight cases; the 45-reference clinical trials dataset was deduplicated correctly by both methods.
- Deduklick found on average 82 more duplicates than information specialists and showed no false-positive cases among all eight datasets; manual deduplication had 11 false-positive references removed unintentionally from one dataset.
- Average manual execution time was 70 min (4200 s); Deduklick averaged below 1 min, with 18,314 references deduplicated in 109 s and 45 references in 2 s.
Methods
- Eight real-life systematic search datasets from University of Bern libraries, average 8,078 references (largest 18,314; smallest 45); six contained pooled publication and clinical trial metadata.
- Manual deduplication used EndNote’s duplicate identification with a multistep, rule-based process and manual eye validation; Deduklick used NLP normalization, Levenshtein-distance similarity scoring, clustering, and rule-based metadata comparisons, with selected threshold θp ≥ 78.
- Gold-standard datasets were created by validating additional duplicates and erroneously removed unique references using low threshold θg ≥ 58; performance was measured by recall, precision, F1, and execution time, excluding flowchart/report generation and system roundtrip.
Limitations
- Stated: only eight EndNote-derived datasets were tested; validating additional databases and testing other datasets are required to explore Deduklick’s full potential.
- Evident: authors QH, NB, and PA work for Risklick AG, the company that developed Deduklick, and the code will not be shared; data are available only upon reasonable request.
- Evident: the benchmark/gold standard depends on manual deduplication plus validation, and the manual comparator used EndNote-specific import order and configurations.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
6 Valid · 1 Not valid · 0 Not checked · 44 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“When used alone, EndNote’s sensitivity for deduplication ranges from 51 to 57% [ 7 , 10 ]”
What the sources say
- Partly supportedcites [7] Rathbone J, Carter M, Hoffmann T, Glasziou P. Better duplicate detection for systematic reviewers: evaluation of systematic review assistant-deduplication module. Syst Rev. 2015;4(6). https://doi.org/10.1186/2046-4053-4-6. sourceExperimental — accuracy not yet measured
The Results report 51.2% sensitivity for EndNote’s default deduplication on one dataset, supporting the lower end of the claimed range. The Validation results and Discussion report 63%, 73%, and 64% on three other datasets; this paper therefore does not support 57% as the upper end of EndNote’s sensitivity range. (LLM, conf=0.89)
- Partly supportedcites [10] Mckeown S, Mir ZM. Considerations for conducting systematic reviews: evaluating the performance of different methods for de-duplicating references. Syst Rev. 2021;10(38). https://doi.org/10.1186/s13643-021-01583-y. sourceExperimental — accuracy not yet measured
The Results (Table 3; page not provided) report 57% sensitivity for EndNote X9 using its default de-duplication settings. They do not report 51% sensitivity or a 51–57% range for EndNote, so this source supports only the upper end of the cited range. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“requiring a median of five researchers and around 40 weeks of work to reach submission [ 3 , 4 ]”
What the sources say
- Supportedcites [3] Clark J, Glasziou P, Del Mar C, Bannach-Brown A, Stehlik P, Scott AM. A full systematic review was completed in 2 weeks using automation tools: a case study. J Clin Epidemiol. 2020;121:81–90. source
The Background states that systematic reviews require a median of five researchers and 41 weeks to submit to a journal. The Discussion repeats both benchmarks. The supplied passages do not give page numbers. (LLM, conf=0.93)
- Partly supportedcites [4] Borah R, Brown AW, Capers PL, Kaiser KA. Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry. BMJ Open. 2017;7(2):e012545. sourceExperimental — accuracy not yet measured
The Abstract Results report a mean of five authors per review, not a median of five researchers. The Results give a mean of 67.3 weeks from registered start to publication, not around 40 weeks to submission; the Abstract's 42-week figure applies only to funded reviews. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“such as accidentally removing unique references—a phenomenon that introduces bias [ 9 , 10 ]”
What the sources say
- Partly supportedcites [9] Hair K, Bahor Z, Macleod M, Liao J, Sena ES. The Automated Systematic Search Deduplicator (ASySD): a rapid, open-source, interoperable tool to remove duplicate citations in biomedical systematic reviews. 2021. https://doi.org/10.1101/2021.05.04.442412 sourceExperimental — accuracy not yet measured
The Background (page unavailable) says that incorrectly removing nonduplicate citations reduces the accuracy and reproducibility of systematic reviews. The Human error section also identifies accidental deletion as a possible reason citations were removed. The paper does not specifically say that such removal introduces bias. (LLM, conf=0.82)
- Partly supportedcites [10] Mckeown S, Mir ZM. Considerations for conducting systematic reviews: evaluating the performance of different methods for de-duplicating references. Syst Rev. 2021;10(38). https://doi.org/10.1186/s13643-021-01583-y. sourceExperimental — accuracy not yet measured
The Discussion says that removing references falsely identified as duplicates may cause eligible studies to be missed and introduce bias into syntheses. The claim states this as an unconditional consequence of removing any unique reference, whereas the paper frames the risk around eligible studies and uses “may.” (LLM, conf=0.86)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“although deduplication of references is essential to ensure the quality of SRs, there is currently no universal method to do so”
What the sources say
- Partly supportedcites [5] Bramer WM, Rethlefsen ML, Kleijnen J, Franco OH. Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study. Syst Rev. 2017;6(245). https://doi.org/10.1186/s13643-017-0644-y. sourceExperimental — accuracy not yet measured
The Methods (Development of search strategies; page not provided) say researchers received deduplicated references for screening, and Data analysis compares counts before and after deduplication. The paper thus treats deduplication as part of its review workflow, but these passages do not establish that it is essential to ensuring systematic-review quality. (LLM, conf=0.82)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On average, deduplication with Deduklick required less than a minute on a development machine with 6-CPU-cores and 32 GB of RAM.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
25 more claims
Create a free account to see all 51 claims and every source verdict.
- Probably not a substantial claim. 8
- It may state two separate claims as one. 15
- The quote only partly states this claim. 2
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“For all datasets presented in Table 4 , we observed an identical distribution of the references before and after deduplication.”
“In two datasets (healthy aging and healthy lifestyle), we observed a larger shift among the first two ranks, which can be explained by the order of import of the datasets into EndNote (therefore, a human factor).”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“We present the numbers of additionally validated true duplicates and erroneously removed unique references for the two manually deduplicated datasets in Table 2 .”
“We validated and confirmed false positives from each of the eight datasets. We summarize the gold standard cross-validation outcome in Table 2 .”
Citations
Cites
The PRISMA 2020 statement: An updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n71 · Cited by 108,283
A typology of reviews: an analysis of 14 review types and associated methodologies
Grant, M. J., & Booth, A. (2009) · Health Information & Libraries Journal · doi:10.1111/j.1471-1842.2009.00848.x · Cited by 10,817 · cited 1×
“Systematic reviews (SRs) and meta-analyses aim to find and synthesize the available evidence within the specific scope of a research question while also striving to minimize bias [1, 2].”
BackgroundRethlefsen, M. L., Kirtley, S., Waffenschmidt, S., Ayala, A. P., Moher, D., Page, M. J., Koffel, J. B., PRISMA-S Group, Blunt, H., Brigham, T., et al. (2021) · Systematic Reviews · doi:10.1186/s13643-020-01542-z · Cited by 3,714
Rethlefsen, M. L., Kirtley, S., Waffenschmidt, S., Ayala, A. P., Moher, D., Page, M. J., & Koffel, J. (2019) · doi:10.31219/osf.io/sfc38 · Cited by 3,696
Bramer, W. M., Rethlefsen, M. L., Kleijnen, J., & Franco, O. H. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0644-y · Cited by 2,114 · cited 3×
“Furthermore, although deduplication of references is essential to ensure the quality of SRs, there is currently no universal method to do so, and the timeintensive task is still mostly performed manually [5].”
BackgroundMethodDe-duplication of database search results for systematic reviews in EndNote
Bramer, W. M., Giustini, D., de Jonge, G. B., Holland, L., & Bekhuis, T. (2016) · Journal of the Medical Library Association : JMLA · doi:10.3163/1536-5050.104.3.014 · Cited by 2,103 · cited 4×
“For example, EndNote [13] is a widely used reference management tool [14] (among others such as Mendeley, RefWorks, and Zotero) offering an integrated deduplication feature, which usually demands careful, multistep configuration procedures [12, 15, 16].”
BackgroundMethodInfluentialBorah, R., Brown, A. W., Capers, P. L., & Kaiser, K. A. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-012545 · Cited by 737 · cited 2×
“Such analyses are time and resource intensive, requiring a median of five researchers and around 40 weeks of work to reach submission [3, 4].”
BackgroundA full systematic review was completed in 2 weeks using automation tools: a case study
Clark, J., Glasziou, P., Del Mar, C., Bannach-Brown, A., Stehlik, P., & Scott, A. M. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.01.008 · Cited by 687 · cited 3×
“Such analyses are time and resource intensive, requiring a median of five researchers and around 40 weeks of work to reach submission [3, 4].”
BackgroundMethodrevtools: An R package to support article screening for evidence synthesis
Westgate, M. J. (2019) · Research Synthesis Methods · doi:10.1002/jrsm.1374 · Cited by 261
McKeown, S., & Mir, Z. M. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01583-y · Cited by 251 · cited 3×
“When used alone, EndNote’s sensitivity for deduplication ranges from 51 to 57% [7, 10].”
BackgroundMethod
Cited by
Lieberum, J.-L., Toews, M., Metzendorf, M.-I., Heilmeyer, F., Siemens, W., Haverkamp, C., Böhringer, D., Meerpohl, J. J., & Eisele-Metzger, A. (2025) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2025.111746 · Cited by 148
Guidance on terminology, application, and reporting of citation searching: the TARCiS statement
Hirt, J., Nordhausen, T., Fürst, T., Ewald, H., & Appenzeller‐Herzog, C. (2024) · BMJ · doi:10.1136/bmj-2023-078384 · Cited by 179
Schwab, T. C., Perrig, L., Göller, P. C., la Hoz, F. F. G. D., Lahousse, A. P., Minder, B., Günther, G., Efthimiou, O., Omar, S. V., Egger, M., et al. (2024) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(24)00263-9 · Cited by 130
Self-citationAutomation of duplicate record detection for systematic reviews: Deduplicator
Forbes, C., Greenwood, H., Carter, M., & Clark, J. M. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02619-9 · Cited by 135
“In research conducted by Borissov et al. [26] Deduklick achieved an impressive mean recall of 99.51% with 100% precision.”
InfluentialSearching for studies: A guide to information retrieval for Campbell systematic reviews
MacDonald, H. R., Comer, C., Foster, M., Labelle, P., Marsalis, S. E., Nyhan, K., Premji, Z., Rogers, M., Splenda, R., Stansfield, C., et al. (2024) · Campbell Systematic Reviews · doi:10.1002/cl2.1433 · Cited by 151
“Review teams may turn to published studies for guidance on best practices for removing duplicates such as considering free tools (Guimarães et al., 2022), comparing reference managers with screening tools (McKeown & Mir, 2021) or turning to newer de ‐ duplication algorithms (Borissov et al., 2022).”
Background
Metadata
- Authors
- Borissov, N., Haas, Q., Minder, B., Kopp-Heim, D., von Gernler, M., Janka, H., Teodoro, D., & Amini, P.
- Affiliations
- University of Bern (CH)
- University of Geneva (CH)
- Corresponding author
- Amini, Poorya
- Venue
- Systematic Reviews
- Published
- 2022-08-17
- Volume
- 11
- Issue
- 1
- Pages
- 172-172
- ISSN
- 2046-4053 (ISSN-L)
- PMC
- PMC9382798
- OpenAlex
- W4292262803
- Cited by
- 116
- References
- 20
- Citations by year
- 2026: 31 · 2025: 40 · 2024: 31 · 2023: 13 · 2022: 1
- FWCI
- 4.18
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Systematic Reviews (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Archive ouverte UNIGE (University of Geneva) (publishedVersion, other-oa)
- Bern Open Repository and Information System (University of Bern) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Archive ouverte UNIGE (University of Geneva) (submittedVersion, cc-by)
- Funders
- Innosuisse - Schweizerische Agentur für Innovationsförderung (41013.1 IP-ICT)
- Topics
- Physical Sciences › Computer Science › Information Systems › Data Quality and Management
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- duplicate detection, citation management, EndNote, natural language processing, metadata normalization, similarity scores, information professionals
- MeSH terms
- Systematic Reviews as Topic, Algorithms, Artificial Intelligence, Humans, Natural Language Processing, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Reducing systematic review burden using Deduklick: a novel, automated, reliable, and explainable deduplication algorithm to foster medical research. https://app.cochat.ai/papers/10.1186/s13643-022-02045-9
- The paper
- Borissov, N., Haas, Q., Minder, B., Kopp-Heim, D., von Gernler, M., Janka, H., Teodoro, D., & Amini, P. (2022). Reducing systematic review burden using Deduklick: a novel, automated, reliable, and explainable deduplication algorithm to foster medical research. Systematic Reviews. https://doi.org/10.1186/s13643-022-02045-9
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.