Evaluation of the citation matching algorithms of CWTS and i FQ in comparison to the W eb of science
Abstract
The results of bibliometric studies provided by bibliometric research groups, for example, the C entre for S cience and T echnology Studies (CWTS) and the I nstitute for R esearch I nformation and Q uality A ssurance (i FQ), are often used in the process of research assessment. Their databases use W eb of Science (WoS) citation data, which they match according to their own matching algorithms—in the case of CWTS for standard usage in their studies and in the case of i FQ on an experimental basis. Because the problem of nonmatched citations in the WoS persists due to inaccuracies in the references or inaccuracies introduced in the data extraction process, it is important to ascertain how well these inaccuracies are rectified in these citation matching algorithms. This article evaluates the algorithms of CWTS and i FQ in comparison to the WoS in a quantitative and a qualitative analysis. The analysis builds upon the method and the manually verified corpus of a previous study. The algorithm of CWTS performs best, closely followed by that of i FQ. The WoS algorithm still performs quite well (F1 score: 96.41%), but shows deficits in matching references containing inaccuracies. An additional problem is posed by incorrectly provided cited reference information in source articles by the WoS.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
WoS and Scopus have specific deficiencies including inconsistencies of journal coverage, inaccuracy of subject classification schemes, missing citation information or citation links, incorrect and missing DOI numbers, duplicate entries, and inconsistent publication dates.
“Meanwhile, other authors discussed more specific deficiencies of the DBs, such as inconsistencies of journal coverage [63], the accuracy of subject classification schemes [64], missing citation information or citation links [40,65,66], incorrect and missing DOI numbers [53,67-69], duplicate entries [30], and inconsistent publication dates [53].”
From Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World (2021) · cites this paper as [65]
A considerable portion of mistakes in publication metadata is caused by incorrect spelling due to language-specific characters, which may be incorrectly identified by optical character recognition systems.
“A considerable portion of mistakes in the publication metadata is caused by incorrect spelling due to the presence of language specific characters (e.g., letters with diacritics), which may be incorrectly identified by optical character recognition (OCR) systems [65,70,71].”
From Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World (2021) · cites this paper as [65]
Some pre-existing errors introduced by authors may be corrected by the publisher in the manuscript editing stage or by the databases during data import.
“some pre-existing errors that are introduced by authors may be corrected by the publisher in the manuscript editing state or by the DBs during the data import [65].”
From Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World (2021) · cites this paper as [65]
The main consequence of all errors in publication metadata is missing or incorrect links between citing and cited papers in the database, which leads to missing citations.
“since the main consequence of all errors in publication metadata is missing or incorrect links between citing and cited papers in the DB, which leads to missing citations [61,62,65,66].”
From Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World (2021) · cites this paper as [65]
Cites
Meho, L. I., & Yang, K. (2007) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20677 · Cited by 974 · see citing works
Comparing bibliometric statistics obtained from the Web of Science and Scopus
Archambault, É., Campbell, D., Gingras, Y., & Larivière, V. (2009) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.21062 · Cited by 833 · see citing works
van Raan, A. F. J. (2005) · Scientometrics · doi:10.1007/s11192-005-0008-6 · Cited by 760 · see citing works
Meho, L. I., & Rogers, Y. (2008) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20874 · Cited by 383 · see citing works
The impact-factors debate: the ISI's uses and limits
Moed, H. F. (2002) · Nature · doi:10.1038/415731a · Cited by 289 · see citing works
Web of Science, Scopus and Google Scholar: A content comprehensiveness comparison
S. Adriaanse, L., & Rensleigh, C. (2013) · The Electronic Library · doi:10.1108/el-12-2011-0174 · Cited by 284 · see citing works
Citation Analysis in Research Evaluation (Information Science & Knowledge Management)
Moed, H. F. (2005) · Springer eBooks · Cited by 206 · see citing works
Data sources for performing citation analysis: an overview
Neuhaus, C., & Daniel, H. (2008) · Journal of Documentation · doi:10.1108/00220410810858010 · Cited by 159 · see citing works
Proceedings of the 17th International conference on science and technology indicators
Larivière, V. (2012) · Cited by 100 · see citing works
Possible inaccuracies occurring in citation analysis
Moed, H. F., & Vriens, M. (1989) · Journal of Information Science · doi:10.1177/016555158901500205 · Cited by 69 · see citing works
Errors in Bibliographic Citations: A Continuing Problem
Sweetland, J. H. (1989) · The Library Quarterly · doi:10.1086/602160 · Cited by 64 · see citing works
Great Expectatrics: Great Papers, Great Journals, Great Econometrics
Chang, C., McAleer, M. J., & Oxley, L. (2011) · Econometric Reviews · doi:10.1080/07474938.2011.586614 · Cited by 37 · see citing works
Franceschini, F., Maisano, D. A., & Mastrogiacomo, L. (2013) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.22898 · Cited by 33 · see citing works
The unification of institutional addresses applying parametrized finite-state graphs (P-FSG)
Gálvez, C., & Anegón, F. D. M. (2006) · Scientometrics · doi:10.1007/s11192-006-0156-3 · Cited by 29 · see citing works
Standardizing formats of corporate source data
Gálvez, C., & Anegón, F. D. M. (2007) · Scientometrics · doi:10.1007/s11192-007-0101-0 · Cited by 25 · see citing works
Reference Accuracy: Best Practices for Making the Links
Meyer, C. A. (2008) · Journal of Electronic Publishing · doi:10.3998/3336451.0011.206 · Cited by 22 · see citing works
Scientists’ referencing (mis)behavior revealed by the dissemination network of referencing errors
Liang, L., Zhong, Z., & Rousseau, R. (2014) · Scientometrics · doi:10.1007/s11192-014-1275-x · Cited by 21 · see citing works
Data accuracy in bibliometric data sources and its impact on citation matching
Olensky, M. (2015) · edoc Publication server (Humboldt University of Berlin) · doi:10.18452/17122 · Cited by 12 · see citing works
Self-citationResearch quality evaluation: comparing citation counts considering bibliometric database errors
Franceschini, F., Maisano, D. A., & Mastrogiacomo, L. (2013) · Quality & Quantity · doi:10.1007/s11135-013-9979-1 · Cited by 11 · see citing works
"The Delphic Oracle" - An Analysis of Potential Error Sources in Bibliographic Databases
Tunger, D., Haustein, S., Ruppert, L., Luca, G., & Unterhalt, S. (2010) · JuSER (Forschungszentrum Jülich) · Cited by 6 · see citing works
Flagged first, then by citations
5 of the 25 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World
Pranckutė, R. (2021) · Publications · doi:10.3390/publications9010012 · Cited by 2,696 · see citing works
A review of the literature on citation impact indicators
Waltman, L. (2016) · Journal of Informetrics · doi:10.1016/j.joi.2016.02.007 · Cited by 1,356 · see citing works
Visser, M., van Eck, N. J., & Waltman, L. (2021) · Quantitative Science Studies · doi:10.1162/qss_a_00112 · Cited by 865 · see citing works
Empirical Analysis and Classification of Database Errors in Scopus and Web of Science
Franceschini, F., Maisano, D., & Mastrogiacomo, L. (2016) · Journal of Informetrics · doi:10.1016/j.joi.2016.07.003 · Cited by 225 · see citing works
Reference coverage analysis of OpenAlex compared to Web of Science and Scopus
Culbert, J. H., Hobert, A., Jahn, N., Haupka, N., Schmidt, M. A., Donner, P., & Mayr, P. (2025) · Scientometrics · doi:10.1007/s11192-025-05293-3 · Cited by 179 · see citing works
Self-citationMeasuring Science: Basic Principles and Application of Advanced Bibliometrics
van Raan, A. F. J. (2019) · Springer handbooks · doi:10.1007/978-3-030-02511-3_10 · Cited by 166 · see citing works
Šubelj, L., van Eck, N. J., & Waltman, L. (2016) · PLoS ONE · doi:10.1371/journal.pone.0154404 · Cited by 142 · see citing works
Thor, A., Marx, W., Loet Leydesdorff, A., & Bornmann, L. (2016) · Journal of Informetrics · doi:10.1016/j.joi.2016.02.005 · Cited by 135 · see citing works
The museum of errors/horrors in Scopus
Franceschini, F., Maisano, D. A., & Mastrogiacomo, L. (2016) · Journal of Informetrics · doi:10.1016/j.joi.2015.11.006 · Cited by 121 · see citing works
Overton: A bibliometric database of policy document citations
Szomszor, M., & Adie, E. A. (2022) · Quantitative Science Studies · doi:10.1162/qss_a_00204 · Cited by 120 · see citing works
Zahedi, Z., & Costas, R. (2018) · PLoS ONE · doi:10.1371/journal.pone.0197326 · Cited by 108 · see citing works
Marx, W., & Bornmann, L. (2016) · Scientometrics · doi:10.1007/s11192-016-2111-2 · Cited by 78 · see citing works
Accuracy of citation data in Web of Science and ScopusPreprint
van Eck, N. J., & Waltman, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.07011 · Cited by 71 · see citing works
Biomedical text mining for research rigor and integrity: tasks, challenges, directions
Kilicoglu, H. (2017) · Briefings in Bioinformatics · doi:10.1093/bib/bbx057 · Cited by 61 · see citing works
Thor, A., Bornmann, L., Marx, W., & Mutz, R. (2018) · Scientometrics · doi:10.1007/s11192-018-2733-7 · Cited by 65 · see citing works
PageRank-based prediction of award-winning researchers and the impact of citations
Fiala, D., & Tutoky, G. (2017) · Journal of Informetrics · doi:10.1016/j.joi.2017.09.008 · Cited by 51 · see citing works
Thinking (in) complexity: (In) definitions and (mis)conceptions
de Melo, A. T., Caves, L. S. D., Dewitt, A., Clutton, E., Macpherson, R., & Garnett, P. (2019) · Systems Research and Behavioral Science · doi:10.1002/sres.2612 · Cited by 40 · see citing works
Effect of publication month on citation impact
Donner, P. (2018) · Journal of Informetrics · doi:10.1016/j.joi.2018.01.012 · Cited by 38 · see citing works
Bibliometrics beyond citations: introducing mention extraction and analysis
Petrovich, E., Verhaegh, S., Bös, G., Cristalli, C., Dewulf, F., van Gemert, T., & IJdens, N. (2024) · Scientometrics · doi:10.1007/s11192-024-05116-x · Cited by 34 · see citing works
Do Scopus and WoS Correct “Old” Omitted Citations?
Franceschini, F., Maisano, D., & Mastrogiacomo, L. (2016) · Scientometrics · doi:10.1007/s11192-016-1867-8 · Cited by 31 · see citing works
New features of CitedReferencesExplorer (CRExplorer)
Thor, A., Marx, W., Loet Leydesdorff, A., & Bornmann, L. (2016) · Scientometrics · doi:10.1007/s11192-016-2082-3 · Cited by 30 · see citing works
ARTIREV: An Integrated Bibliometric Tool to Efficiently Conduct Quality Literature Reviews
Walsh, I., Renaud, A., Medina, M. J., Baudet, C., & Mourmant, G. (2023) · Systèmes d information & management · doi:10.3917/sim.224.0005 · Cited by 23 · see citing works
Silva, F. S. V., Schulz, P. A., & Noyons, E. (2018) · Scientometrics · doi:10.1007/s11192-018-2967-4 · Cited by 22 · see citing works
Dubois, S., & Walsh, I. (2017) · M n gement · doi:10.3917/mana.205.0435 · Cited by 17 · see citing works
Haunschild, R., Hug, S. E., Brändle, M. P., & Bornmann, L. (2017) · Scientometrics · doi:10.1007/s11192-017-2567-8 · Cited by 16 · see citing works
Mobile Forensics: A Bibliometric Analysis
Gill, J., Okere, I., HaddadPajouh, H., & Dehghantanha, A. (2018) · Advances in information security · doi:10.1007/978-3-319-73951-9_15 · Cited by 15 · see citing works
TimeRank: A dynamic approach to rate scholars using citations
Franceschet, M., & Colavizza, G. (2017) · Journal of Informetrics · doi:10.1016/j.joi.2017.09.003 · Cited by 14 · see citing works
Determinants of Renewable Energy Technology Deployment: A Systematic Review
Kunskaja, S., & Pažėraitė, A. (2025) · Sustainability · doi:10.3390/su172310538 · Cited by 13 · see citing works
Sampling issues in bibliometric analysis: Response to discussants
Williams, R. A., & Bornmann, L. (2016) · Journal of Informetrics · doi:10.1016/j.joi.2016.09.013 · Cited by 12 · see citing works
Data inaccuracy quantification and uncertainty propagation for bibliometric indicators
Donner, P. (2024) · Research Evaluation · doi:10.1093/reseval/rvae047 · Cited by 12 · see citing works
Maisano, D. A., Mastrogiacomo, L., Ferrara, L., & Franceschini, F. (2025) · Scientometrics · doi:10.1007/s11192-025-05244-y · Cited by 11 · see citing works
Developing indicators on Open Access by combining evidence from diverse data sourcesPreprint
van Leeuwen, T. N., Meijer, I., Yegros, A., & Costas, R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.02827 · Cited by 11 · see citing works
Zagovora, O., Ulloa, R., Weller, K., & Flöck, F. (2021) · Quantitative Science Studies · doi:10.1162/qss_a_00171 · Cited by 10 · see citing works
Bautista‐Puig, N., Aleixo, A. M., Leal, S. C. H., Azeiteiro, U. M., & Costas, R. (2021) · Frontiers in Sustainability · doi:10.3389/frsus.2021.620743 · Cited by 9 · see citing works
Schmidt, M. A., Rimmert, C., Stephen, D. M., Lenke, C., Donner, P., Gärtner, S., Taubert, N., Bausenwein, T., & Stahlschmidt, S. (2025) · Quantitative Science Studies · doi:10.1162/qss.a.20 · Cited by 8 · see citing works
Self-citationBoutracheh, H., & Moumen, A. (2025) · International Journal of Hydrogen Energy · doi:10.1016/j.ijhydene.2025.151526 · Cited by 6 · see citing works
Bautista‐Puig, N., Mañana-Rodríguez, J., & Serrano-López, A. E. (2021) · Scientometrics · doi:10.1007/s11192-021-03939-6 · Cited by 5 · see citing works
The structural role of the core literature in history
Colavizza, G. (2017) · Scientometrics · doi:10.1007/s11192-017-2550-4 · Cited by 3 · see citing works
Gunashekar, S., Parks, S., Calero‐Medina, C., Visser, M. S., van Honk, J., & Wooding, S. (2015) · RAND Corporation eBooks · doi:10.7249/rr1363 · Cited by 3 · see citing works
EXmatcher: Combining Features Based on Reference Strings and Segments to Enhance Citation MatchingPreprint
Ghavimi, B., Otto, W., & Mayr, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.04484 · Cited by 3 · see citing works
Assessing data quality in citation analysis: A case study of web of science and Crossref
Rong, G., Chen, Y., Koch, T., & Honda, K. (2026) · Journal of Informetrics · doi:10.1016/j.joi.2026.101775 · Cited by 2 · see citing works
Errors in document-type classification: a focus on engineering publications and their publishers
Maisano, D., Ferrara, L., & Franceschini, F. (2025) · Scientometrics · doi:10.1007/s11192-025-05444-6 · Cited by 1 · see citing works
Widespread reference missingness disparities in open scholarly metadata
Zhou, H., Liang, L., & Acuna, D. E. (2025) · Quantitative Science Studies · doi:10.1162/qss.a.400 · Cited by 1 · see citing works
Compliance of New Scientific Platforms with Formal Requirements for Scientific Publications
Gorbunova, A., Zassoursky, I. I., & Trishchenko, N. D. (2020) · Scholarly Research and Information · doi:10.24108/2658-3143-2020-3-2-3-207-218 · Cited by 1 · see citing works
Uncertain research country rankings. Should we continue producing uncertain rankings?
Rodríguez‐Navarro, A. (2025) · Journal of Data and Information Science · doi:10.2478/jdis-2025-0030 · Cited by 2 · see citing works
(2023) · Data Science and Informetrics · doi:10.59494/dsi.2023.2.1 · Cited by 1 · see citing works
The structural role of the core literature in historyPreprint
Colavizza, G. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.02204 · Cited by 1 · see citing works
Refcat: The Internet Archive Scholar Citation GraphPreprint
Czygan, M., Holzmann, H., & Newbold, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.06595 · Cited by 1 · see citing works
A Magyar Pedagógia folyóirat tudománymetriai és tartalmi elemzése
Nagy, G. (2019) · doi:10.14232/phd.10233 · Cited by 1 · see citing works
Evaluation of unique identifiers used for citation linking
Madsen, H. H., Madsen, D., & Gauffriau, M. (2016) · F1000Research · doi:10.12688/f1000research.8913.1
Madsen, H. H., Madsen, D., & Gauffriau, M. (2016) · F1000Research · doi:10.12688/f1000research.8913.2
Adding software to package management systems can increase their citation by 280%Preprint
Jalili, V., Clements, D., Grüning, B. A., Blankenberg, D. J., & Goecks, J. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.11.16.385211
Can the World’s Research Ecosystem Be Openly Indexed?
Ho, L. (2025) · Katina : · doi:10.1146/katina-01062025-1
Analysis of research and development trends of solar photovoltaic buildings based on CiteSpace
Mo, R., Luo, H., Li, Y., Liu, Y., Li, Y., Kang, W., & Meng, X. (2025) · Journal of Asian Architecture and Building Engineering · doi:10.1080/13467581.2025.2602263
- Authors
- Olensky, M., Schmidt, M., & van Eck, N. J.
- Affiliations
- Humboldt-Universität zu Berlin (DE)
- Institute for Research Information and Quality Assurance (DE)
- Centre for Science and Technology Studies (CWTS) Leiden University P.O. Box 905 Leiden 2300 AX The Netherlands
- Corresponding author
- Olensky, Marlies
- Schmidt, Marion
- Venue
- Journal of the Association for Information Science and Technology
- Publisher
- Wiley
- Published
- 2015-08-25
- Volume
- 67
- Issue
- 10
- Pages
- 2550-2564
- ISSN
- 2330-1635 (ISSN-L)
- 2330-1643
- OpenAlex
- W1920360620
- Cited by
- 56
- References
- 25
- Citations by year
- 2026: 1 · 2025: 10 · 2024: 2 · 2023: 2 · 2022: 1 · 2021: 7 · 2020: 2 · 2019: 5 · 2018: 6 · 2017: 7 · 2016: 12 · 2015: 1
- FWCI
- 7.91
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of the Association for Information Science and Technology (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Leiden Repository (Leiden University) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
- Physical Sciences › Computer Science › Information Systems › Web visibility and informetrics
- Physical Sciences › Computer Science › Information Systems › Library Science and Information Systems
- Keywords
- Web of Science, CWTS, bibliometric databases, research evaluation, citation accuracy, bibliometric analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
