The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Coverage of highly-cited documents in Google Scholar, Web of Science, and Scopus: a multidisciplinary comparison
Open accessPublished version of a preprint
Abstract
This study explores the extent to which bibliometric indicators based on counts of highly-cited documents could be affected by the choice of data source. The initial hypothesis is that databases that rely on journal selection criteria for their document coverage may not necessarily provide an accurate representation of highly-cited documents across all subject areas, while inclusive databases, which give each document the chance to stand on its own merits, might be better suited to identify highly-cited documents. To test this hypothesis, an analysis of 2515 highly-cited documents published in 2006 that Google Scholar displays in its Classic Papers product is carried out at the level of broad subject categories, checking whether these documents are also covered in Web of Science and Scopus, and whether the citation counts offered by the different sources are similar. The results show that a large fraction of highly-cited documents in the Social Sciences and Humanities (8.6–28.2%) are invisible to Web of Science and Scopus. In the Natural, Life, and Health Sciences the proportion of missing highly-cited documents in Web of Science and Scopus is much lower. Furthermore, in all areas, Spearman correlation coefficients of citation counts in Google Scholar, as compared to Web of Science and Scopus citation counts, are remarkably strong (.83–.99). The main conclusion is that the data about highly-cited documents available in the inclusive database Google Scholar does indeed reveal significant coverage deficiencies in Web of Science and Scopus in several areas of research. Therefore, using these selective databases to compute bibliometric indicators based on counts of highly-cited documents might produce biased assessments in poorly covered areas.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Bibliometrics: The Leiden Manifesto for Research Metrics
Hicks, D., Wouters, P., Waltman, L., de Rijcke, S., & Rafols, I. (2015) · Nature · doi:10.1038/520429a · Cited by 2,828
Meho, L. I., & Yang, K. (2007) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20677 · Cited by 974
Halevi, G., Moed, H., & Bar-Ilan, J. (2017) · Journal of Informetrics · doi:10.1016/j.joi.2017.06.005 · Cited by 488
To what extent is inclusion in the Web of Science an indicator of journal ‘quality’?
Chavarro, D., Ràfols, I., & Tang, P. (2018) · Research Evaluation · doi:10.1093/reseval/rvy001 · Cited by 115
García‐Pérez, M. A. (2010) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.21372 · Cited by 101
Research works printed but not linked: 35
- Acharya, A., Verstak, A., Suzuki, H., Henderson, S., Iakhiaev, M., Lin, C. C. Y., & Shetty, N. (2014). Rise of the rest: The growing impact of non-elite journals. Retrieved June 19, 2018, from http://arxiv.org/abs/1410.2217 . · arXiv
- Archambault, É., Vignola-Gagné, É., Côté, G., Larivière, V., & Gingrasb, Y. (2006). Benchmarking scientific output in the social sciences and humanities: The limits of existing databases. Scientometrics, 68(3), 329–342. https://doi.org/10.1007/s11192-006-0115-z . · DOI · Paper page
- Bar-Ilan, J. (2008). Which h-index?—A comparison of WoS, Scopus and Google Scholar. Scientometrics, 74(2), 257–271. https://doi.org/10.1007/s11192-008-0216-y . · DOI · Paper page
- Bar-Ilan, J. (2010). Citations to the “Introduction to informetrics” indexed by WOS, Scopus and Google Scholar. Scientometrics, 82(3), 495–506. https://doi.org/10.1007/s11192-010-0185-9 . · DOI · Paper page
- Bornmann, L., & Leydesdorff, L. (2018). Count highly-cited papers instead of papers with h citations: Use normalized citation counts and compare “like with like”! Scientometrics, 115(2), 1119–1123. https://doi.org/10.1007/s11192-018-2682-1 . · DOI
- Bornmann, L., & Marx, W. (2014). How to evaluate individual researchers working in the natural and life sciences meaningfully? A proposal of methods based on percentiles of citations. Scientometrics, 98(1), 487–509. https://doi.org/10.1007/s11192-013-1161-y . · DOI
- Bornmann, L., Marx, W., Schier, H., Rahm, E., Thor, A., & Daniel, H.-D. (2009). Convergent validity of bibliometric Google Scholar data in the field of chemistry—Citation counts for papers that were accepted by Angewandte Chemie International Edition or rejected but published elsewhere, using Google Scholar, Science Citation Index, Scopus, and Chemical Abstracts. Journal of Informetrics, 3(1), 27–35. https://doi.org/10.1016/j.joi.2008.11.001 . · DOI
- De Solla Price, D. (1976). A general theory of bibliometric and other cumulative advantage processes. Journal of the American Society for Information Science, 27(5), 292–306. https://doi.org/10.1002/asi.4630270505 . · DOI · Paper page
- de Winter, J. C. F., Zadpoor, A. A., & Dodou, D. (2013). The expansion of Google Scholar versus Web of Science: A longitudinal study. Scientometrics, 98(2), 1547–1565. https://doi.org/10.1007/s11192-013-1089-2 . · DOI · Paper page
- Delgado López-Cózar, E., Martín-Martín, A., & Orduna-Malea, E. (2017). Classic papers: déjà vu, a step further in the bibliometric exploitation of Google Scholar (EC3’s Working Papers No. 24). Retrieved June 19, 2018, from https://arxiv.org/abs/1706.09258 . · arXiv
- Delgado López-Cózar, E., Orduna-Malea, E., & Martín-Martín, A. (2018). Google Scholar as a data source for research assessment. In W. Glaenzel, H. Moed, U. Schmoch & M. Thelwall (Eds.), Springer handbook of science and technology indicators. Berlin: Springer.
- Delgado López-Cózar, E., Robinson-García, N., & Torres-Salinas, D. (2014). The Google scholar experiment: How to index false papers and manipulate bibliometric indicators. Journal of the Association for Information Science and Technology, 65(3), 446–454. https://doi.org/10.1002/asi.23056 . · DOI · Paper page
- Else, H. (2018). How I scraped data from Google Scholar. Nature. https://doi.org/10.1038/d41586-018-04190-5 . · DOI
- Frandsen, T. F., & Nicolaisen, J. (2008). Intradisciplinary differences in database coverage and the consequences for bibliometric research. Journal of the American Society for Information Science and Technology, 59(10), 1570–1581. https://doi.org/10.1002/asi.20817 . · DOI
- Harzing, A.-W. (2013). A preliminary test of Google Scholar as a source for citation data: A longitudinal study of Nobel prize winners. Scientometrics, 94(3), 1057–1075. https://doi.org/10.1007/s11192-012-0777-7 . · DOI · Paper page
- Kousha, K., & Thelwall, M. (2008). Sources of Google Scholar citations outside the Science Citation Index: A comparison between four science disciplines. Scientometrics, 74(2), 273–294. https://doi.org/10.1007/s11192-008-0217-x . · DOI · Paper page
- Levine-Clark, M., & Gil, E. L. (2008). A comparative citation analysis of Web of Science, Scopus, and Google Scholar. Journal of Business & Finance Librarianship, 14(1), 32–46. https://doi.org/10.1080/08963560802176348 . · DOI
- Leydesdorff, L., Bornmann, L., Mutz, R., & Opthof, T. (2011). Turning the tables on citation analysis one more time: Principles for comparing sets of documents. Journal of the American Society for Information Science and Technology, 62(7), 1370–1381. https://doi.org/10.1002/asi.21534 . · DOI · Paper page
- Martín-Martín, A., Costas, R., van Leeuwen, T., & Delgado López-Cózar, E. (2018). Evidence of Open Access of scientific publications in Google Scholar: A large-scale analysis. https://doi.org/10.17605/osf.io/k54uv . · DOI
- Martín-Martín, A., Orduna-Malea, E., Ayllón, J. M., & Delgado López-Cózar, E. (2016). A two-sided academic landscape: snapshot of highly-cited documents in Google Scholar (1950–2013). Revista Española de Documentacion Cientifica, 39(4), e149. https://doi.org/10.3989/redc.2016.4.1405 . · DOI
- Martín-Martín, A., Orduña-Malea, E., Ayllón, J. M., & Delgado-López-Cózar, E. (2014). Does Google Scholar contain all highly cited documents (1950–2013)? (EC3 Working Papers No. 19). Retrieved June 19, 2018, from http://arxiv.org/abs/1410.8464 . · arXiv
- Martín-Martín, A., Orduna-Malea, E., & Delgado López-Cózar, E. (2018). Data and code for: Coverage of highly-cited documents in Google Scholar, Web of Science, and Scopus: A multidisciplinary comparison. https://doi.org/10.17605/OSF.IO/DNQZK . · doi:10.17605/osf.io/dnqzk
- Martin-Martin, A., Orduna-Malea, E., Harzing, A.-W., & Delgado López-Cózar, E. (2017). Can we use Google Scholar to identify highly-cited documents? Journal of Informetrics, 11(1), 152–163. https://doi.org/10.1016/j.joi.2016.11.008 . · DOI · Paper page
- Mingers, J., & Lipitakis, E. A. E. C. G. (2010). Counting the citations: A comparison of Web of Science and Google Scholar in the field of business and management. Scientometrics, 85(2), 613–625. https://doi.org/10.1007/s11192-010-0270-0 . · DOI · Paper page
- Moed, H. F., Bar-Ilan, J., & Halevi, G. (2016). A new methodology for comparing Google Scholar and Scopus. Journal of Informetrics, 10(2), 533–551. https://doi.org/10.1016/j.joi.2016.04.017 . · DOI
- Mongeon, P., & Paul-Hus, A. (2016). The journal coverage of Web of Science and Scopus: A comparative analysis. Scientometrics, 106(1), 213–228. https://doi.org/10.1007/s11192-015-1765-5 . · DOI · Paper page
- Orduna-Malea, E., Ayllón, J. M., Martín-Martín, A., & Delgado López-Cózar, E. (2015). Methods for estimating the size of Google Scholar. Scientometrics, 104(3), 931–949. https://doi.org/10.1007/s11192-015-1614-6 . · DOI
- Orduña-Malea, E., Martín-Martín, A., Ayllón, J. M., & Delgado López-Cózar, E. (2016). La revolución Google Scholar: Destapando la caja de Pandora académica. Granada: Universidad de Granada.
- Orduna-Malea, E., Martín-Martín, A., & Delgado López-Cózar, E. (2017). Google Scholar as a source for scholarly evaluation: A bibliographic review of database errors. Revista Española de Documentación Científica, 40(4), e185. https://doi.org/10.3989/redc.2017.4.1500 . · DOI
- Rahimi, S., & Chandrakumar, V. (2014). A comparison of citation coverage of traditional and web citation databases in medical science. Malaysian Journal of Library and Information Science, 19(3), 1–11. Retrieved June 19, 2018, from http://jice.um.edu.my/index.php/MJLIS/article/view/1779 .
- Sud, P., & Thelwall, M. (2014). Evaluating altmetrics. Scientometrics, 98(2), 1131–1143. https://doi.org/10.1007/s11192-013-1117-2 . · DOI · Paper page
- Thelwall, M. (2017). Three practical field normalised alternative indicator formulae for research evaluation. Journal of Informetrics, 11(1), 128–151. https://doi.org/10.1016/j.joi.2016.12.002 . · DOI
- Thelwall, M., & Fairclough, R. (2017). The accuracy of confidence intervals for field normalised indicators. Journal of Informetrics, 11(2), 530–540. https://doi.org/10.1016/j.joi.2017.03.004 . · DOI
- van Leeuwen, T. N., Moed, H. F., Tijssen, R. J. W., Visser, M. S., & Van Raan, A. F. J. (2001). Language biases in the coverage of the Science Citation Index and its consequences for international comparisons of national research performance. Scientometrics, 51(1), 335–346. https://doi.org/10.1023/A:1010549719484 . · DOI · Paper page
- Vine, R. (2006). Google Scholar. Journal of the Medical Library Association, 94(1), 97. Retrieved from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1324783/ .
Cited by
Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World
Pranckutė, R. (2021) · Publications · doi:10.3390/publications9010012 · Cited by 2,696
The Journal Coverage of Web of Science, Scopus and Dimensions: A Comparative Analysis
Singh, V. K., Singh, P., Karmakar, M., Leta, J., & Mayr, P. (2021) · Scientometrics · doi:10.1007/s11192-021-03948-5 · Cited by 2,342
Martín-Martín, A., Orduna-Malea, E., Thelwall, M., & Delgado López-Cózar, E. (2018) · Journal of Informetrics · doi:10.1016/j.joi.2018.09.002 · Cited by 1,990
Self-citationVisser, M., van Eck, N. J., & Waltman, L. (2021) · Quantitative Science Studies · doi:10.1162/qss_a_00112 · Cited by 865
Gusenbauer, M. (2019) · Scientometrics · doi:10.1007/s11192-018-2958-5 · Cited by 687
Metadata
- Authors
- Martín-Martín, A., Orduna-Malea, E., & Delgado López-Cózar, E.
- Venue
- Scientometrics
- Published
- 2018-06-26
- Volume
- 116
- Issue
- 3
- Pages
- 2175-2188
- ISSN
- 0138-9130 (ISSN-L)
- 1588-2861
- OpenAlex
- W2950619306
- Cited by
- 195
- References
- 5
- FWCI
- 10.17
- Type
- journal article
- Language
- en
- Open access
- green
- Licence
- www.springer.com/tdm (tdm)
- Funders
- Ministerio de Educación, Cultura y Deporte (FPU2013/05863)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Coverage of highly-cited documents in Google Scholar, Web of Science, and Scopus: a multidisciplinary comparison. https://app.cochat.ai/papers/10.1007/s11192-018-2820-9
- The paper
- Martín-Martín, A., Orduna-Malea, E., & Delgado López-Cózar, E. (2018). Coverage of highly-cited documents in Google Scholar, Web of Science, and Scopus: a multidisciplinary comparison. Scientometrics. https://doi.org/10.1007/s11192-018-2820-9
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.