Considerations for conducting systematic reviews: evaluating the performance of different methods for de-duplicating references
Open accessThe DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
Abstract
Background: Systematic reviews involve searching multiple bibliographic databases to identify eligible studies. As this type of evidence synthesis is increasingly pursued, the use of various electronic platforms can help researchers improve the efficiency and quality of their research. We examined the accuracy and efficiency of commonly used electronic methods for flagging and removing duplicate references during this process. Methods: A heterogeneous sample of references was obtained by conducting a similar topical search in MEDLINE, Embase, Cochrane Central Register of Controlled Trials, and PsycINFO databases. References were de-duplicated via manual abstraction to create a benchmark set. The default settings were then used in Ovid multifile search, EndNote desktop, Mendeley, Zotero, Covidence, and Rayyan to de-duplicate the sample of references independently. Using the benchmark set as reference, the number of false-negative and false-positive duplicate references for each method was identified, and accuracy, sensitivity, and specificity were determined. Results: We found that the most accurate methods for identifying duplicate references were Ovid, Covidence, and Rayyan. Ovid and Covidence possessed the highest specificity for identifying duplicate references, while Rayyan demonstrated the highest sensitivity. Conclusion: This study reveals the strengths and weaknesses of commonly used de-duplication methods and provides strategies for improving their performance to avoid unintentionally removing eligible studies and introducing bias into systematic reviews. Along with availability, ease-of-use, functionality, and capability, these findings are important to consider when researchers are selecting database platforms and supporting software programs for conducting systematic reviews.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“When used alone, EndNote’s sensitivity for deduplication ranges from 51 to 57% [ 7 , 10 ]”
From Reducing systematic review burden using Deduklick: a novel, automated, reliable, and explainable deduplication algorithm to foster medical research (2022) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“such as accidentally removing unique references—a phenomenon that introduces bias [ 9 , 10 ]”
From Reducing systematic review burden using Deduklick: a novel, automated, reliable, and explainable deduplication algorithm to foster medical research (2022) · cites this paper as [10]
Experimental — accuracy not yet measuredIn the citing paper
“Automatic reference deduplication algorithms have shown acceptable performance results [27], but inconsistent evaluation results remain across datasets and in the way their performance is assessed across the literature [27][28][29].”
From A narrative review of recent tools and innovations toward automating living systematic reviews and evidence syntheses (2023) · cites this paper as [29]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Rayyan-a web and mobile app for systematic reviews.
Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0384-4 · Cited by 26,092
Cochrane Handbook for Systematic Reviews of Interventions
(2019) · doi:10.1002/9781119536604 · Cited by 14,828
THE USE OF CONFIDENCE OR FIDUCIAL LIMITS ILLUSTRATED IN THE CASE OF THE BINOMIAL
CLOPPER, C. J., & PEARSON, E. S. (1934) · Biometrika · doi:10.1093/biomet/26.4.404 · Cited by 5,003
De-duplication of database search results for systematic reviews in EndNote
Bramer, W. M., Giustini, D., de Jonge, G. B., Holland, L., & Bekhuis, T. (2016) · Journal of the Medical Library Association : JMLA · doi:10.3163/1536-5050.104.3.014 · Cited by 2,103
The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta‐analyses
IOANNIDIS, J. P. A. (2016) · The Milbank Quarterly · doi:10.1111/1468-0009.12210 · Cited by 1,483
Page, M. J., Shamseer, L., Altman, D. G., Tetzlaff, J., Sampson, M., Tricco, A. C., Catalá-López, F., Li, L., Reid, E. K., Sarkis-Onofre, R., et al. (2016) · PLOS Medicine · doi:10.1371/journal.pmed.1002028 · Cited by 768 · cited 1×
“Unable to retrieve full-text publication for 10 false-positive duplicate references within EndNote X9 Unable to retrieve full-text publication for 2 false-positive duplicate references within Rayyan McKeown and Mir Systematic Reviews (2021) 10:38 Page 5 of 8”
BackgroundStress Ulcer Prophylaxis in Critically III Patients: Resolving Discordant Meta-analyses
Cook, D. J. (1996) · JAMA · doi:10.1001/jama.1996.03530280060038 · Cited by 608
Ouweneel, D. M., Schotborgh, J. V., Limpens, J., Sjauw, K. D., Engström, A. E., Lagrand, W. K., Cherpanath, T. G. V., Driessen, A. H. G., de Mol, B. A. J. M., & Henriques, J. P. S. (2016) · Intensive Care Medicine · doi:10.1007/s00134-016-4536-8 · Cited by 538
Kellermeyer, L., Harnke, B., & Knight, S. (2018) · Journal of the Medical Library Association · doi:10.5195/jmla.2018.513 · Cited by 380
Searching for studies: a guide to information retrieval for Campbell systematic reviews
Kugley, S., Wade, A., Thomas, J., Mahood, Q., Jørgensen, A. K., Hammerstrøm, K., & Sathe, N. (2017) · Campbell Systematic Reviews · doi:10.4073/cmg.2016.1 · Cited by 371
Research works printed but not linked: 1
- Covidence systematic review software. Veritas Health Innovation, Melbourne, Australia. https://www.covidence.org. Accessed 17 Mar 2020.
Cited by
Artificial intelligence in systematic reviews: promising when appropriately used
van Dijk, S. H. B., Brusse-Keizer, M. G. J., Bucsán, C. C., van der Palen, J., Doggen, C. J. M., & Lenferink, A. (2023) · BMJ Open · doi:10.1136/bmjopen-2023-072254 · Cited by 256
Assay of renewable energy transition: A systematic literature review
Bhattarai, U., Maraseni, T., & Apan, A. (2022) · Science of The Total Environment · doi:10.1016/j.scitotenv.2022.155159 · Cited by 173
Borissov, N., Haas, Q., Minder, B., Kopp-Heim, D., von Gernler, M., Janka, H., Teodoro, D., & Amini, P. (2022) · Systematic Reviews · doi:10.1186/s13643-022-02045-9 · Cited by 116
“When used alone, EndNote’s sensitivity for deduplication ranges from 51 to 57% [7, 10].”
BackgroundMethodAutomation of duplicate record detection for systematic reviews: Deduplicator
Forbes, C., Greenwood, H., Carter, M., & Clark, J. M. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02619-9 · Cited by 150
AI-based learning style detection in adaptive learning systems: a systematic literature review
Ezzaim, A., Dahbi, A., Aqqal, A., & Haidine, A. (2025) · Journal of Computers in Education · doi:10.1007/s40692-024-00328-9 · Cited by 109
Metadata
- Authors
- McKeown, S., & Mir, Z. M.
- Affiliations
- Queen's University (CA)
- Corresponding author
- McKeown, Sandra
- Venue
- Systematic Reviews
- Published
- 2021-01-23
- Volume
- 10
- Issue
- 1
- Pages
- 38-38
- ISSN
- 2046-4053 (ISSN-L)
- PMC
- PMC7827976
- OpenAlex
- W3121686920
- Cited by
- 251
- References
- 26
- Citations by year
- 2026: 52 · 2025: 53 · 2024: 66 · 2023: 46 · 2022: 31 · 2021: 3
- FWCI
- 1.62
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by-sa
- creativecommons.org/licenses/by/4.0/ (tdm)
- creativecommons.org/licenses/by/4.0/ (vor)
- Locations
- Systematic Reviews (publishedVersion, cc-by-sa)
- Europe PMC (PubMed Central) (publishedVersion, other-oa)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Keywords
- data deduplication, systematic review screening, bibliographic database searching, Ovid, Covidence, Rayyan, EndNote, Mendeley, Zotero, software accuracy, false positives and false negatives, MEDLINE, Embase, PsycINFO
- MeSH terms
- Systematic Reviews as Topic, Humans, Databases, Bibliographic, MEDLINE, Information Storage and Retrieval
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Considerations for conducting systematic reviews: evaluating the performance of different methods for de-duplicating references. https://app.cochat.ai/papers/10.1186/s13643-021-01583-y
- The paper
- McKeown, S., & Mir, Z. M. (2021). Considerations for conducting systematic reviews: evaluating the performance of different methods for de-duplicating references. Systematic Reviews. https://doi.org/10.1186/s13643-021-01583-y
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.