Google Scholar is not enough to be used alone for systematic reviews
Open accessAbstract
BACKGROUND: Google Scholar (GS) has been noted for its ability to search broadly for important references in the literature. Gehanno et al. recently examined GS in their study: 'Is Google scholar enough to be used alone for systematic reviews?' In this paper, we revisit this important question, and some of Gehanno et al.'s other findings in evaluating the academic search engine. METHODS: The authors searched for a recent systematic review (SR) of comparable size to run search tests similar to those in Gehanno et al. We selected Chou et al. (2013) contacting the authors for a list of publications they found in their SR on social media in health. We queried GS for each of those 506 titles (in quotes ""), one by one. When GS failed to retrieve a paper, or produced too many results, we used the allintitle: command to find papers with the same title. RESULTS: Google Scholar produced records for ~95% of the papers cited by Chou et al. (n=476/506). A few of the 30 papers that were not in GS were later retrieved via PubMed and even regular Google Search. But due to its different structure, we could not run searches in GS that were originally performed by Chou et al. in PubMed, Web of Science, Scopus and PsycINFO®. Identifying 506 papers in GS was an inefficient process, especially for papers using similar search terms. CONCLUSIONS: Has Google Scholar improved enough to be used alone in searching for systematic reviews? No. GS' constantly-changing content, algorithms and database structure make it a poor choice for systematic reviews. Looking for papers when you know their titles is a far different issue from discovering them initially. Further research is needed to determine when and how (and for what purposes) GS can be used alone. Google should provide details about GS' database coverage and improve its interface (e.g., with semantic search filters, stored searching, etc.). Perhaps then it will be an appropriate choice for systematic reviews.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Until now, studies have largely examined the suitability of search systems for certain scholarly tasks for individual systems or by comparing a few systems.
“Until now, studies have largely examined the suitability of search systems for certain scholarly tasks for individual systems or by comparing a few systems.33, 50, 95”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Google Scholar: the pros and the cons
Jacsó, P. (2005) · Online Information Review · doi:10.1108/14684520510598066 · Cited by 383 · cited 1× · see citing works
“But GS was already deemed unsuitable for literature reviews due to its limited search (filtering and qualifiers) functionality; its inability to draw on the power of the MeSH vocabulary (used in MEDLINE/PubMed) was cited as a critical flaw [14,15].”
BackgroundIs the coverage of google scholar enough to be used alone for systematic reviews
Gehanno, J.-F., Rollin, L., & Darmoni, S. (2013) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-13-7 · Cited by 357 · see citing works
Web 2.0 for Health Promotion: Reviewing the Current Evidence
Chou, W.-Y. S., Prestin, A., Lyons, C., & Wen, K.-Y. (2013) · American Journal of Public Health · doi:10.2105/ajph.2012.301071 · Cited by 309 · see citing works
Comparing test searches in PubMed and Google Scholar
Shultz, M. (2007) · Journal of the Medical Library Association JMLA · doi:10.3163/1536-5050.95.4.442 · Cited by 231 · see citing works
Kousha, K., & Thelwall, M. A. (2007) · Scientometrics · doi:10.1007/s11192-008-0217-x · Cited by 169 · see citing works
The Depth and Breadth of Google Scholar: An Empirical Study
Neuhaus, C., Neuhaus, E., Asher, A., & Wrede, C. (2006) · portal: Libraries and the Academy · doi:10.1353/pla.2006.0026 · Cited by 156 · see citing works
How Google is changing medicine
Giustini, D. (2005) · BMJ · doi:10.1136/bmj.331.7531.1487 · Cited by 118 · see citing works
Self-citationComparison of PubMed and Google Scholar Literature Searches
Anders, M., & Evans, D. P. (2010) · Respiratory Care · doi:10.4187/respcare.10550578 · Cited by 117 · see citing works
Medical literature searches: a comparison of P ub M ed and G oogle S cholar
Nourbakhsh, E., Nugent, R., Wang, H., Çevik, C., & NUGENT, K. M. (2012) · Health Information & Libraries Journal · doi:10.1111/j.1471-1842.2012.00992.x · Cited by 113 · see citing works
Google Scholar: A source for clinicians?
Henderson, J. (2005) · Canadian Medical Association Journal · doi:10.1503/cmaj.050404 · Cited by 76 · see citing works
A look at Google Scholar, PubMed, and Scirus: comparisons and recommendations
Giustini, D., & Barsky, E. (2005) · Journal of the Canadian Health Libraries Association / Journal de l Association de bilbiothèques de la santé du Canada · doi:10.5596/c05-030 · Cited by 60 · see citing works
Self-citationWalters, W. H. (2011) · portal Libraries and the Academy · doi:10.1353/pla.2011.0042 · Cited by 51 · see citing works
Hightower, C., & Caldwell, C. (2010) · Issues in Science and Technology Librarianship · doi:10.29173/istl2545 · Cited by 35 · see citing works
The excitement of Google Scholar, the worry of Google Print
Banks, M. A. (2005) · Biomedical Digital Libraries · doi:10.1186/1742-5581-2-2 · Cited by 24 · see citing works
Mastrangelo, G., Fadda, E., Rossi, C. R., Zamprogno, E., Buja, A., & Cegolon, L. (2010) · BMC Research Notes · doi:10.1186/1756-0500-3-131 · Cited by 21 · see citing works
Wilson, V. (2007) · DOAJ (DOAJ: Directory of Open Access Journals) · Cited by 8 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- [7]Vine R 2006 Google Scholar. J Med Libr Assoc. 94(1), 97- 99
- [9]Vine R 2006 Google Scholar. J Med Libr Assoc. 94(1), 97- 99
Cited by
Sorted by citations
The role of google scholar in evidence reviews and its applicability to grey literature searching
Haddaway, N. R., Collins, A. M., Coughlin, D., & Kirk, S. (2015) · PLOS ONE · doi:10.1371/journal.pone.0138237 · Cited by 2,387 · see citing works
Gusenbauer, M., & Haddaway, N. R. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1378 · Cited by 2,379 · see citing works
“Until now, studies have largely examined the suitability of search systems for certain scholarly tasks for individual systems or by comparing a few systems.(33,50,95) Our methods allowed a comprehensive review of many different search systems.”
MethodGeorge, S., Duran, N., & Norris, K. (2014) · American Journal of Public Health · doi:10.2105/ajph.2013.301706 · Cited by 1,603 · see citing works
Halevi, G., Moed, H., & Bar-Ilan, J. (2017) · Journal of Informetrics · doi:10.1016/j.joi.2017.06.005 · Cited by 488 · see citing works
Boeker, M., Vach, W., & Motschall, E. (2013) · BMC Medical Research Methodology · doi:10.1186/1471-2288-13-131 · Cited by 225 · see citing works
On the Effectiveness of Supplemental Instruction
Dawson, P., van der Meer, J., Skalicky, J., & Cowley, K. (2014) · Review of Educational Research · doi:10.3102/0034654314540007 · Cited by 230 · see citing works
Systematic searching for environmental evidence using multiple tools and sources
Livoreil, B., Glanville, J., Haddaway, N. R., Bayliss, H., Bethel, A., de Lachapelle, F. F., Robalino, S., Savilaakso, S., Zhou, W., Petrokofsky, G., et al. (2017) · Environmental Evidence · doi:10.1186/s13750-017-0099-6 · Cited by 156 · see citing works
Bramer, W. M., Giustini, D., Kramer, B. M., & Anderson, P. (2013) · Systematic Reviews · doi:10.1186/2046-4053-2-115 · Cited by 139 · see citing works
Self-citationFit for purpose: perspectives on rapid reviews from end-user interviews
Hartling, L., Guise, J.-M., Hempel, S., Featherstone, R., Mitchell, M. D., Motu’apuaka, M. L., Robinson, K. A., Schoelles, K., Totten, A., Whitlock, E., et al. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0425-7 · Cited by 65 · see citing works
Linder, S. K., Kamath, G. R., Pratt, G. F., Saraykar, S. S., & Volk, R. J. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.10.008 · Cited by 63 · see citing works
Briscoe, S. (2015) · BMC Research Notes · doi:10.1186/s13104-015-1079-y · Cited by 57 · see citing works
Fuller, T., Peters, J., Pearson, M., & Anderson, R. (2014) · American Journal of Public Health · doi:10.2105/ajph.2014.302195 · Cited by 30 · see citing works
- Authors
- Giustini, D., & Kamel Boulos, M. N.
- Affiliations
- University of British Columbia (CA)
- ORCID
- Giustini, Dean
- Venue
- Online Journal of Public Health Informatics
- Publisher
- University of Illinois Libraries
- Published
- 2013-06
- Volume
- 5
- Issue
- 2
- Pages
- 214-214
- ISSN
- 1947-2579 (ISSN-L)
- PMC
- PMC3733758
- OpenAlex
- W2094444890
- Cited by
- 187
- References
- 16
- Citations by year
- 2026: 16 · 2025: 20 · 2024: 21 · 2023: 18 · 2022: 19 · 2021: 15 · 2020: 13 · 2019: 12 · 2018: 5 · 2017: 9 · 2016: 10 · 2015: 15 · 2014: 10 · 2013: 4
- FWCI
- 2.63
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Online Journal of Public Health Informatics (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
- Keywords
- Google Scholar, search engine evaluation, database coverage, search efficiency, academic search engines
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
