Student or educator? Sign up with your school email and get 6 months of Pro free.
Cited by 195

The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.

Coverage of highly-cited documents in Google Scholar, Web of Science, and Scopus: a multidisciplinary comparison

Open access
Martín-Martín, Alberto, Orduna-Malea, Enrique, Delgado López-Cózar, Emilio·2018·Scientometrics

Published version of a preprint

Abstract

This study explores the extent to which bibliometric indicators based on counts of highly-cited documents could be affected by the choice of data source. The initial hypothesis is that databases that rely on journal selection criteria for their document coverage may not necessarily provide an accurate representation of highly-cited documents across all subject areas, while inclusive databases, which give each document the chance to stand on its own merits, might be better suited to identify highly-cited documents. To test this hypothesis, an analysis of 2515 highly-cited documents published in 2006 that Google Scholar displays in its Classic Papers product is carried out at the level of broad subject categories, checking whether these documents are also covered in Web of Science and Scopus, and whether the citation counts offered by the different sources are similar. The results show that a large fraction of highly-cited documents in the Social Sciences and Humanities (8.6–28.2%) are invisible to Web of Science and Scopus. In the Natural, Life, and Health Sciences the proportion of missing highly-cited documents in Web of Science and Scopus is much lower. Furthermore, in all areas, Spearman correlation coefficients of citation counts in Google Scholar, as compared to Web of Science and Scopus citation counts, are remarkably strong (.83–.99). The main conclusion is that the data about highly-cited documents available in the inclusive database Google Scholar does indeed reveal significant coverage deficiencies in Web of Science and Scopus in several areas of research. Therefore, using these selective databases to compute bibliometric indicators based on counts of highly-cited documents might produce biased assessments in poorly covered areas.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s11192-018-2820-9

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 5 references

Research works printed but not linked: 35

  1. Acharya, A., Verstak, A., Suzuki, H., Henderson, S., Iakhiaev, M., Lin, C. C. Y., & Shetty, N. (2014). Rise of the rest: The growing impact of non-elite journals. Retrieved June 19, 2018, from http://arxiv.org/abs/1410.2217 . · arXiv
  2. Archambault, É., Vignola-Gagné, É., Côté, G., Larivière, V., & Gingrasb, Y. (2006). Benchmarking scientific output in the social sciences and humanities: The limits of existing databases. Scientometrics, 68(3), 329–342. https://doi.org/10.1007/s11192-006-0115-z . · DOI · Paper page
  3. Bar-Ilan, J. (2008). Which h-index?—A comparison of WoS, Scopus and Google Scholar. Scientometrics, 74(2), 257–271. https://doi.org/10.1007/s11192-008-0216-y . · DOI · Paper page
  4. Bar-Ilan, J. (2010). Citations to the “Introduction to informetrics” indexed by WOS, Scopus and Google Scholar. Scientometrics, 82(3), 495–506. https://doi.org/10.1007/s11192-010-0185-9 . · DOI · Paper page
  5. Bornmann, L., & Leydesdorff, L. (2018). Count highly-cited papers instead of papers with h citations: Use normalized citation counts and compare “like with like”! Scientometrics, 115(2), 1119–1123. https://doi.org/10.1007/s11192-018-2682-1 . · DOI
  6. Bornmann, L., & Marx, W. (2014). How to evaluate individual researchers working in the natural and life sciences meaningfully? A proposal of methods based on percentiles of citations. Scientometrics, 98(1), 487–509. https://doi.org/10.1007/s11192-013-1161-y . · DOI
  7. Bornmann, L., Marx, W., Schier, H., Rahm, E., Thor, A., & Daniel, H.-D. (2009). Convergent validity of bibliometric Google Scholar data in the field of chemistry—Citation counts for papers that were accepted by Angewandte Chemie International Edition or rejected but published elsewhere, using Google Scholar, Science Citation Index, Scopus, and Chemical Abstracts. Journal of Informetrics, 3(1), 27–35. https://doi.org/10.1016/j.joi.2008.11.001 . · DOI
  8. De Solla Price, D. (1976). A general theory of bibliometric and other cumulative advantage processes. Journal of the American Society for Information Science, 27(5), 292–306. https://doi.org/10.1002/asi.4630270505 . · DOI · Paper page
  9. de Winter, J. C. F., Zadpoor, A. A., & Dodou, D. (2013). The expansion of Google Scholar versus Web of Science: A longitudinal study. Scientometrics, 98(2), 1547–1565. https://doi.org/10.1007/s11192-013-1089-2 . · DOI · Paper page
  10. Delgado López-Cózar, E., Martín-Martín, A., & Orduna-Malea, E. (2017). Classic papers: déjà vu, a step further in the bibliometric exploitation of Google Scholar (EC3’s Working Papers No. 24). Retrieved June 19, 2018, from https://arxiv.org/abs/1706.09258 . · arXiv
  11. Delgado López-Cózar, E., Orduna-Malea, E., & Martín-Martín, A. (2018). Google Scholar as a data source for research assessment. In W. Glaenzel, H. Moed, U. Schmoch & M. Thelwall (Eds.), Springer handbook of science and technology indicators. Berlin: Springer.
  12. Delgado López-Cózar, E., Robinson-García, N., & Torres-Salinas, D. (2014). The Google scholar experiment: How to index false papers and manipulate bibliometric indicators. Journal of the Association for Information Science and Technology, 65(3), 446–454. https://doi.org/10.1002/asi.23056 . · DOI · Paper page
  13. Else, H. (2018). How I scraped data from Google Scholar. Nature. https://doi.org/10.1038/d41586-018-04190-5 . · DOI
  14. Frandsen, T. F., & Nicolaisen, J. (2008). Intradisciplinary differences in database coverage and the consequences for bibliometric research. Journal of the American Society for Information Science and Technology, 59(10), 1570–1581. https://doi.org/10.1002/asi.20817 . · DOI
  15. Harzing, A.-W. (2013). A preliminary test of Google Scholar as a source for citation data: A longitudinal study of Nobel prize winners. Scientometrics, 94(3), 1057–1075. https://doi.org/10.1007/s11192-012-0777-7 . · DOI · Paper page
  16. Kousha, K., & Thelwall, M. (2008). Sources of Google Scholar citations outside the Science Citation Index: A comparison between four science disciplines. Scientometrics, 74(2), 273–294. https://doi.org/10.1007/s11192-008-0217-x . · DOI · Paper page
  17. Levine-Clark, M., & Gil, E. L. (2008). A comparative citation analysis of Web of Science, Scopus, and Google Scholar. Journal of Business & Finance Librarianship, 14(1), 32–46. https://doi.org/10.1080/08963560802176348 . · DOI
  18. Leydesdorff, L., Bornmann, L., Mutz, R., & Opthof, T. (2011). Turning the tables on citation analysis one more time: Principles for comparing sets of documents. Journal of the American Society for Information Science and Technology, 62(7), 1370–1381. https://doi.org/10.1002/asi.21534 . · DOI · Paper page
  19. Martín-Martín, A., Costas, R., van Leeuwen, T., & Delgado López-Cózar, E. (2018). Evidence of Open Access of scientific publications in Google Scholar: A large-scale analysis. https://doi.org/10.17605/osf.io/k54uv . · DOI
  20. Martín-Martín, A., Orduna-Malea, E., Ayllón, J. M., & Delgado López-Cózar, E. (2016). A two-sided academic landscape: snapshot of highly-cited documents in Google Scholar (1950–2013). Revista Española de Documentacion Cientifica, 39(4), e149. https://doi.org/10.3989/redc.2016.4.1405 . · DOI
  21. Martín-Martín, A., Orduña-Malea, E., Ayllón, J. M., & Delgado-López-Cózar, E. (2014). Does Google Scholar contain all highly cited documents (1950–2013)? (EC3 Working Papers No. 19). Retrieved June 19, 2018, from http://arxiv.org/abs/1410.8464 . · arXiv
  22. Martín-Martín, A., Orduna-Malea, E., & Delgado López-Cózar, E. (2018). Data and code for: Coverage of highly-cited documents in Google Scholar, Web of Science, and Scopus: A multidisciplinary comparison. https://doi.org/10.17605/OSF.IO/DNQZK . · doi:10.17605/osf.io/dnqzk
  23. Martin-Martin, A., Orduna-Malea, E., Harzing, A.-W., & Delgado López-Cózar, E. (2017). Can we use Google Scholar to identify highly-cited documents? Journal of Informetrics, 11(1), 152–163. https://doi.org/10.1016/j.joi.2016.11.008 . · DOI · Paper page
  24. Mingers, J., & Lipitakis, E. A. E. C. G. (2010). Counting the citations: A comparison of Web of Science and Google Scholar in the field of business and management. Scientometrics, 85(2), 613–625. https://doi.org/10.1007/s11192-010-0270-0 . · DOI · Paper page
  25. Moed, H. F., Bar-Ilan, J., & Halevi, G. (2016). A new methodology for comparing Google Scholar and Scopus. Journal of Informetrics, 10(2), 533–551. https://doi.org/10.1016/j.joi.2016.04.017 . · DOI
  26. Mongeon, P., & Paul-Hus, A. (2016). The journal coverage of Web of Science and Scopus: A comparative analysis. Scientometrics, 106(1), 213–228. https://doi.org/10.1007/s11192-015-1765-5 . · DOI · Paper page
  27. Orduna-Malea, E., Ayllón, J. M., Martín-Martín, A., & Delgado López-Cózar, E. (2015). Methods for estimating the size of Google Scholar. Scientometrics, 104(3), 931–949. https://doi.org/10.1007/s11192-015-1614-6 . · DOI
  28. Orduña-Malea, E., Martín-Martín, A., Ayllón, J. M., & Delgado López-Cózar, E. (2016). La revolución Google Scholar: Destapando la caja de Pandora académica. Granada: Universidad de Granada.
  29. Orduna-Malea, E., Martín-Martín, A., & Delgado López-Cózar, E. (2017). Google Scholar as a source for scholarly evaluation: A bibliographic review of database errors. Revista Española de Documentación Científica, 40(4), e185. https://doi.org/10.3989/redc.2017.4.1500 . · DOI
  30. Rahimi, S., & Chandrakumar, V. (2014). A comparison of citation coverage of traditional and web citation databases in medical science. Malaysian Journal of Library and Information Science, 19(3), 1–11. Retrieved June 19, 2018, from http://jice.um.edu.my/index.php/MJLIS/article/view/1779 .
  31. Sud, P., & Thelwall, M. (2014). Evaluating altmetrics. Scientometrics, 98(2), 1131–1143. https://doi.org/10.1007/s11192-013-1117-2 . · DOI · Paper page
  32. Thelwall, M. (2017). Three practical field normalised alternative indicator formulae for research evaluation. Journal of Informetrics, 11(1), 128–151. https://doi.org/10.1016/j.joi.2016.12.002 . · DOI
  33. Thelwall, M., & Fairclough, R. (2017). The accuracy of confidence intervals for field normalised indicators. Journal of Informetrics, 11(2), 530–540. https://doi.org/10.1016/j.joi.2017.03.004 . · DOI
  34. van Leeuwen, T. N., Moed, H. F., Tijssen, R. J. W., Visser, M. S., & Van Raan, A. F. J. (2001). Language biases in the coverage of the Science Citation Index and its consequences for international comparisons of national research performance. Scientometrics, 51(1), 335–346. https://doi.org/10.1023/A:1010549719484 . · DOI · Paper page
  35. Vine, R. (2006). Google Scholar. Journal of the Medical Library Association, 94(1), 97. Retrieved from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1324783/ .

Cited by

See all 195

Metadata

Authors
Martín-Martín, A., Orduna-Malea, E., & Delgado López-Cózar, E.
Venue
Scientometrics
Published
2018-06-26
Volume
116
Issue
3
Pages
2175-2188
ISSN
  • 0138-9130 (ISSN-L)
  • 1588-2861
OpenAlex
W2950619306
Cited by
195
References
5
FWCI
10.17
Type
journal article
Language
en
Open access
green
Licence
www.springer.com/tdm (tdm)
Funders
Ministerio de Educación, Cultura y Deporte (FPU2013/05863)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Coverage of highly-cited documents in Google Scholar, Web of Science, and Scopus: a multidisciplinary comparison. https://app.cochat.ai/papers/10.1007/s11192-018-2820-9
The paper
Martín-Martín, A., Orduna-Malea, E., & Delgado López-Cózar, E. (2018). Coverage of highly-cited documents in Google Scholar, Web of Science, and Scopus: a multidisciplinary comparison. Scientometrics. https://doi.org/10.1007/s11192-018-2820-9

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.