The DOI matches its Crossref record; 56 of the 58 references it cites were found in a registry or an index, and 1 are not checked yet.
Google Scholar as replacement for systematic literature searches: good relative recall and precision are not enough
Open accessAbstract
BACKGROUND: Recent research indicates a high recall in Google Scholar searches for systematic reviews. These reports raised high expectations of Google Scholar as a unified and easy to use search interface. However, studies on the coverage of Google Scholar rarely used the search interface in a realistic approach but instead merely checked for the existence of gold standard references. In addition, the severe limitations of the Google Search interface must be taken into consideration when comparing with professional literature retrieval tools.The objectives of this work are to measure the relative recall and precision of searches with Google Scholar under conditions which are derived from structured search procedures conventional in scientific literature retrieval; and to provide an overview of current advantages and disadvantages of the Google Scholar search interface in scientific literature retrieval. METHODS: General and MEDLINE-specific search strategies were retrieved from 14 Cochrane systematic reviews. Cochrane systematic review search strategies were translated to Google Scholar search expression as good as possible under consideration of the original search semantics. The references of the included studies from the Cochrane reviews were checked for their inclusion in the result sets of the Google Scholar searches. Relative recall and precision were calculated. RESULTS: We investigated Cochrane reviews with a number of included references between 11 and 70 with a total of 396 references. The Google Scholar searches resulted in sets between 4,320 and 67,800 and a total of 291,190 hits. The relative recall of the Google Scholar searches had a minimum of 76.2% and a maximum of 100% (7 searches). The precision of the Google Scholar searches had a minimum of 0.05% and a maximum of 0.92%. The overall relative recall for all searches was 92.9%, the overall precision was 0.13%. CONCLUSION: The reported relative recall must be interpreted with care. It is a quality indicator of Google Scholar confined to an experimental setting which is unavailable in systematic retrieval due to the severe limitations of the Google Scholar search interface. Currently, Google Scholar does not provide necessary elements for systematic scientific literature retrieval such as tools for incremental query optimization, export of a large number of references, a visual search builder or a history function. Google Scholar is not ready as a professional searching tool for tasks where structured retrieval methodology is necessary.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This research thus fills a need for support in the choice of search system, currently lacking in evidence‐synthesis methodology, and follows calls for comparative studies on the effectiveness of search systems25 or the “need to develop ‘bias profiles’ for search engines.”( 35 , p1193)”
From Which academic search systems are suitable for systematic reviews or meta‐analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources (2020) · cites this paper as [25]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Giustini and Boulos [14] found that 5% of studies from a systematic review could not be identified using specific searches in Google Scholar, whilst Boeker et al. [11] found that up to 34% of studies from 14 systematic reviews were missed.”
From The role of google scholar in evidence reviews and its applicability to grey literature searching (2015) · cites this paper as [11]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Quality criteria have previously been used to evaluate search systems in terms of recall and precision indicating their suitability of search in general and systematic search in particular.”
The cited source says what the claim attributes to it.
In the citing paper
“Our findings support the criticism of Bramer et al,33 Bramer et al,34 and Boeker et al25 and indicate that Google Scholar's coverage and recall is an inadequate reason to use it as principal search system in systematic searches.53”
In the citing paper
“The ability of a reviewer to manipulate their search query depends on the capabilities of a search system and thus they significantly influence recall and precision.25”
In the citing paper
“Nevertheless, overall, Google Scholar's search precision has been found to be significantly lower than 1% for systematic searches.25”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000100 · Cited by 28,164 · cited 1×
“In a continuing discussion and agreement process, the information science and the scientific literature retrieval communities have developed methods and standards which aim for a high level of quality in literature retrieval and reporting [14-18].”
BackgroundCochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048
Cochrane Handbook for Systematic Reviews of Interventions
Tarsilla, M. (2010) · Journal of MultiDisciplinary Evaluation · doi:10.56645/jmde.v6i14.284 · Cited by 5,232
Comparison of PubMed, Scopus, Web of Science, and Google Scholar: strengths and weaknesses
Falagas, M. E., Pitsouni, E. I., Malietzis, G. A., & Pappas, G. (2008) · The FASEB Journal · doi:10.1096/fj.07-9492lsf · Cited by 5,039 · cited 1×
“This technology known as autonomous citation indexing, is also applied to the Web of Knowledge and Scopus, however, with differing results [37,38].”
ResultA catalogue of reporting guidelines for health research
Simera, I., Moher, D., Hoey, J., Schulz, K. F., & Altman, D. G. (2010) · European Journal of Clinical Investigation · doi:10.1111/j.1365-2362.2009.02234.x · Cited by 1,047 · cited 1×
“In a continuing discussion and agreement process, the information science and the scientific literature retrieval communities have developed methods and standards which aim for a high level of quality in literature retrieval and reporting [14-18].”
BackgroundKulkarni, A. V. (2009) · JAMA · doi:10.1001/jama.2009.1307 · Cited by 821 · cited 1×
“This technology known as autonomous citation indexing, is also applied to the Web of Knowledge and Scopus, however, with differing results [37,38].”
ResultDetection of gene pathways with predictive power for breast cancer prognosis
Ma, S., & Kosorok, M. R. (2010) · BMC Bioinformatics · doi:10.1186/1471-2105-11-1 · Cited by 693
An evidence-based practice guideline for the peer review of electronic search strategies
Sampson, M., McGowan, J., Cogo, E., Grimshaw, J., Moher, D., & Lefebvre, C. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2008.10.012 · Cited by 608 · cited 1×
“In a continuing discussion and agreement process, the information science and the scientific literature retrieval communities have developed methods and standards which aim for a high level of quality in literature retrieval and reporting [14-18].”
BackgroundLefebvre, C., Manheimer, E. W., & Glanville, J. (2008) · doi:10.1002/9780470712184.ch6 · Cited by 599
Selecting Studies and Collecting Data
Higgins, J. P., & Deeks, J. J. (2008) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9780470712184.ch7 · Cited by 467 · cited 2×
“Assuming fast reference checking for exclusion of irrelevant references as a first step of manual study selection [41], an experienced information specialist can check up to 1,000 references a day [53].”
Background
Research works printed but not linked: 6
- Heinzl S: Medizinreport: studien im fokus. Medizinisch-wissenschaftliche recherche: Google scholar ist PubMed möglicherweise überlegen. Dtsch Arztebl. 2012, 109: A–2467-
- Ovaska T, Fiander M, Weightman A: BMC medical informatics and decision making – user comments – is the coverage of Google scholar enough to be used alone for systematic reviews. 2013, http://www.biomedcentral.com/1472-6947/13/7/comments ,
- Cochrane Editorial Unit: Methodological expectations of Cochrane intervention reviews (MECIR). 2013, http://www.editorial-unit.cochrane.org/mecir ,
- Google Inc: About Google scholar. 2013, http://www.google.com/intl/en/scholar/about.html ,
- Jasco P: As we may search – comparison of major features of the Web of science, Scopus, and Google scholar citation-based and citation-enhanced databases. Curr Sci. 2005, 89: 1537-1547.
- The Cochrane Collaboration: The Cochrane Library. Independent high-quality evidence for health care decision making. 2013, http://www.thecochranelibrary.com/view/0/index.html ,
Cited by
Sorted by citations
The role of google scholar in evidence reviews and its applicability to grey literature searching
Haddaway, N. R., Collins, A. M., Coughlin, D., & Kirk, S. (2015) · PLOS ONE · doi:10.1371/journal.pone.0138237 · Cited by 2,387
“[11,12], but also as a replacement for traditional academic citation databases and in searches for grey literature in systematic reviews [13,14].”
BackgroundInfluentialGusenbauer, M., & Haddaway, N. R. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1378 · Cited by 2,379
“25 This is not surprising, since our findings show that Google Scholar does not support many of the features required for systematic searches.”
BackgroundResultInfluentialHalevi, G., Moed, H., & Bar-Ilan, J. (2017) · Journal of Informetrics · doi:10.1016/j.joi.2017.06.005 · Cited by 488
Consolidated principles for screening based on a systematic review and consensus process
Dobrow, M. J., Hagens, V., Chafe, R., Sullivan, T., & Rabeneck, L. (2018) · Canadian Medical Association Journal · doi:10.1503/cmaj.171154 · Cited by 359
Wearable sensor-based real-time gait detection: A systematic review
Prasanth, H., Caban, M., Keller, U., Courtine, G., Ijspeert, A., Vallery, H., & von Zitzewitz, J. (2021) · Sensors · doi:10.3390/s21082727 · Cited by 348
Newton, P. M. (2018) · Frontiers in Education · doi:10.3389/feduc.2018.00067 · Cited by 341
Optimal literature search for systematic reviews in surgery
Gooßen, K., Tenckhoff, S., Probst, P., Grummich, K., Mihaljević, A. L., Büchler, M. W., & Diener, M. K. (2017) · Langenbeck s Archives of Surgery · doi:10.1007/s00423-017-1646-x · Cited by 247
Robotic technologies for on-site building construction: A systematic review
Gharbia, M., Chang-Richards, A. Y., Lu, Y., Zhong, R. Y., & Li, H. (2020) · Journal of Building Engineering · doi:10.1016/j.jobe.2020.101584 · Cited by 256
Powered exoskeletons for bipedal locomotion after spinal cord injury
Contreras-Vidal, J. L., Bhagat, N. A., Brantley, J. A., Cruz-Garza, J. G., He, Y., Manley, Q., Nakagome, S., Nathan, K., Tan, S. H., Zhu, F., et al. (2016) · Journal of Neural Engineering · doi:10.1088/1741-2560/13/3/031001 · Cited by 217
“Since Google Scholar restricts search terms to 256 characters (Boeker et al 2013), we added the exoskeleton names to the search terms one at a time, i.e. (ORIGINAL_SEARCH_TERMS) AND (EXOSKELETON_NAME).”
BackgroundBramer, W. M., Giustini, D., & Kramer, B. M. R. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0215-7 · Cited by 202
“Low precision has always been considered a problem in GS [11], but when accounting for actually usable search results (i.”
BackgroundOloyede, A. A., Faruk, N., Noma, N., Tebepah, E., & Nwaulune, A. K. (2023) · Heliyon · doi:10.1016/j.heliyon.2023.e17654 · Cited by 151
“Search engine such as google scholar was not utilized because of its difficulty in extracting the searched literature, and the suitability of google scholar for this type of search has also been questioned by Boeker et al. [20].”
MethodNewton, P. M., & Essex, K. (2023) · Journal of Academic Ethics · doi:10.1007/s10805-023-09485-5 · Cited by 193
“The Google Scholar search interface is limited, and the search returns can include non-research documents search as citations, university policies and handbooks on academic integrity, and multiple versions of papers (Boeker et al., 2013).”
Bramer, W. M., Giustini, D., Kramer, B. M., & Anderson, P. (2013) · Systematic Reviews · doi:10.1186/2046-4053-2-115 · Cited by 139
Main gamification concepts: A systematic mapping study
Rodrigues, L. F., Oliveira, A., & Rodrigues, H. (2019) · Heliyon · doi:10.1016/j.heliyon.2019.e01993 · Cited by 183
“The use of Google Scholar by the scienti fi c community seems convincing enough to promote it as an acceptable and reliable search tool for systematic reviews (Petersen et al., 2008; Bailey et al., 2007; Boeker et al., 2013; Giustini and Barsky, 2005).”
MethodModeling languages in Industry 4.0: an extended systematic mapping study
Wortmann, A., Barais, O., Combemale, B., & Wimmer, M. (2019) · Software & Systems Modeling · doi:10.1007/s10270-019-00757-6 · Cited by 136
Narcissism and Social Networking Behavior: A Meta‐Analysis
Gnambs, T., & Appel, M. (2017) · Journal of Personality · doi:10.1111/jopy.12305 · Cited by 156
A rapid method to increase transparency and efficiency in web-based searches
Haddaway, N. R., Collins, A. M., Coughlin, D., & Kirk, S. (2017) · Environmental Evidence · doi:10.1186/s13750-016-0079-2 · Cited by 129
“We do not advocate the use of search engines such as Google Scholar as alternatives to bibliographic databases in systematic reviews: this has been demonstrated to lack comprehensiveness and repeatability [1].”
MethodStansfield, C., Dickson, K., & Bangpan, M. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0371-9 · Cited by 122
A Cross-Domain Systematic Mapping Study on Software Engineering for Digital Twins
Dalibor, M., Jansen, N., Rumpe⋆, B., Schmalzing, D., Wachtmeister, L., Wimmer, M., & Wortmann, A. (2022) · Journal of Systems and Software · doi:10.1016/j.jss.2022.111361 · Cited by 147
Newton, P. M., & Salvi, A. (2020) · Frontiers in Education · doi:10.3389/feduc.2020.602451 · Cited by 164
“However, all Google Scholar results had to hand screened in real-time since Google Scholar does not have the same functionality as the databases described above; it includes multiple versions of the same papers, and the search interface is limited, making it difficult to accurately quantify and report search results (…”
BackgroundMulti-Criteria Decision-Making Methods to Address Water Allocation Problems: A Systematic Review
Gebre, S. L., Cattrysse, D., & Orshoven, J. V. (2021) · Water · doi:10.3390/w13020125 · Cited by 114
“, tutorials, posters, presentations Search results are not reproducible and reportable International and interdisciplinary coverage (all types of documents) Few sorting options; many nonpeer-reviewed sources Google Unknown [28,29,34]”
BackgroundGnambs, T., & Kaspar, K. (2016) · Assessment · doi:10.1177/1073191115624547 · Cited by 106
Góngora, P., Rocks, S., Fahr, P., Rivero‐Arias, O., & Tsiachristas, A. (2022) · Value in Health · doi:10.1016/j.jval.2022.11.007 · Cited by 118
Crime scripting: A systematic review
Dehghanniri, H., & Borrion, H. (2019) · European Journal of Criminology · doi:10.1177/1477370819850943 · Cited by 102
“Despite criticisms regarding the use of Google Scholar in systematic reviews (see Boeker et al., 2013), we decided to use it to conduct a forward citation search because the main article, Cornish (1994b), was not available in any of the above data sources.”
MethodGebre, S. L., Cattrysse, D., Alemayehu, E., & Orshoven, J. V. (2021) · International Soil and Water Conservation Research · doi:10.1016/j.iswcr.2021.04.005 · Cited by 109
O'Connor, A. M. C., Anderson, K. M., Goodell, C. K., & Sargeant, J. M. (2014) · Zoonoses and Public Health · doi:10.1111/zph.12125 · Cited by 102
“It is important to know the limitations of web-based search engines, such as difficulties reproducing results and difficulties exporting results to reference management software (Boeker et al., 2013).”
BackgroundO'Connor, S. R., Connaghan, J., Maguire, R., Kotronoulas, G., Flannagan, C., Jain, S., Brady, N., & McCaughan, E. (2018) · Patient Education and Counseling · doi:10.1016/j.pec.2018.12.015 · Cited by 88
“Grey literature [18] was identified from http://www.”
Model composition in Model Driven Engineering: A systematic literature review
Abouzahra, A., Sabraoui, A., & Afdel, K. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106316 · Cited by 87
Predatory publications in evidence syntheses
Ross‐White, A., Godfrey, C. M., Sears, K., & Wilson, R. A. (2019) · Journal of the Medical Library Association JMLA · doi:10.5195/jmla.2019.491 · Cited by 82
“Other researchers have similarly expressed caution about the use of Google Scholar as a source in systematic reviews and meta-analyses [19, 20], as well as its use as a research tool in general [21, 22].”
BackgroundA systematic scoping review of the Social Vulnerability Index as applied to natural hazards
Painter, M. A., Shah, S. H., Damestoit, G. C., Khalid, F., Prudencio, W., Chisty, M. A., Tormos‐Aponte, F., & Wilhelmi, O. V. (2024) · Natural Hazards · doi:10.1007/s11069-023-06378-z · Cited by 90
Gusenbauer, M., & Haddaway, N. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1457 · Cited by 88
“2,11 Most academics are unaware of the different types of searching that they use on a day-to-day basis.”
BackgroundThe Role of AI in Serious Games and Gamification for Health: Scoping Review
Tolks, D., Schmidt, J., & Kühn, S. (2023) · JMIR Serious Games · doi:10.2196/48258 · Cited by 87
“However, the results from Google Scholar were not precise enough for inclusion in a review, which is consistent with the results of several studies [34-36].”
ResultYuan, Z., Wen, B., He, C., Zhou, J., Zhou, Z., & Xu, F. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph19116572 · Cited by 83
“Google Scholar was excluded due to the singularity of the search function, the lack of categorical tags, and imprecise search results [69,70].”
BackgroundCampo, K. X., Teper, T., Eaton, C., Shipman, A. M., Bhatia, G., & Mesmer, B. L. (2022) · Systems Engineering · doi:10.1002/sys.21644 · Cited by 76
TheoryOn: A Design Framework and System for Unlocking Behavioral Knowledge Through Ontology Learning
Li, J., Larsen, K. R. T., & Abbasi, A. (2020) · MIS Quarterly · doi:10.25300/misq/2020/15323 · Cited by 71
“Boeker et al. (2013) found that across 14 existing systematic studies, the precision of Google Scholar was 0.13%.”
BackgroundInfluentialSteinau, S., Marrella, A., Andrews, K., Leotta, F., Mecella, M., & Reichert, M. (2018) · Software & Systems Modeling · doi:10.1007/s10270-018-0695-0 · Cited by 67
Christ, A. A., Sieber, V., Grob, U., & Praetorius, A. (2022) · Studies In Educational Evaluation · doi:10.1016/j.stueduc.2022.101209 · Cited by 92
Linder, S. K., Kamath, G. R., Pratt, G. F., Saraykar, S. S., & Volk, R. J. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.10.008 · Cited by 63
Briscoe, S. (2015) · BMC Research Notes · doi:10.1186/s13104-015-1079-y · Cited by 57
Toward a Formalization of the Process to Select IMIA Yearbook Best Papers
Brigitte Séroussi, A., Griffon, N., Kerdelhué, G., Jaulent, M., Jacques Bouaud, A., & Lamy, J. B. (2014) · Methods of Information in Medicine · doi:10.3414/me14-01-0031 · Cited by 61
“The lack of structure in references [30 –32], e.g. the truncation of authors’ lists and journal names, prevented matching entries from other databases with a near-zero error rate.”
BackgroundThe Implementation of Internet Interventions for Depression: A Scoping Review
Drozd, F., Vaskinn, L., Bergsund, H. B., Haga, S. M., Slinning, K., & Bjørkli, C. A. (2016) · Journal of Medical Internet Research · doi:10.2196/jmir.5670 · Cited by 67
“Google Scholar was only used for additional searches, as it is not a traditional scientific database and has been found unsuitable for systematic literature search [30].”
Larsen, K. R. T., Hovorka, D. S., Dennis, A. R., & West, J. D. (2019) · Journal of the Association for Information Systems · doi:10.17705/1jais.00556 · Cited by 64
“Boeker et al. (2013) provided perhaps one of the most striking conclusions on the use of Google Scholar (GS) for systematic literature review, showing an atrocious precision of 0.0013 when evaluating its ability to find a set of manuscripts using the search strategy used in systematic literature…”
BackgroundInfluentialKorać, S., Saliterer, I., & Weigand, B. (2018) · International Public Management Journal · doi:10.1080/10967494.2018.1430086 · Cited by 60
“Although this search generated a high number of results (mainly due to the limited filters, marginal applicability of Boolean search, the non-applicability of truncation), it enhanced risk of non-replicability due to changing algorithms and database structures (see Boeker, Vach, and Motschall 2013).”
BackgroundHartmann, D., Wang, S. M., Pohlmann, L., & Berendt, B. (2025) · Journal of Computational Social Science · doi:10.1007/s42001-025-00381-z · Cited by 59
“24 For example, a study might find that people who have many connections in a social network are more likely to adopt a certain behavior.”
BackgroundLast-Mile Strategies for Urban Freight Delivery: A Systematic Review
Lyons, T., & McDonald, N. (2022) · Transportation Research Record Journal of the Transportation Research Board · doi:10.1177/03611981221103596 · Cited by 55
“We excluded Google Scholar as the opacity of its search algorithms and the surfeit of results returned would run contrary to the purpose of a systematic review (7).”
ResultEysenbach, J. T. G. (2001) · Medical Informatics and the Internet in Medicine · doi:10.1080/14639230110075459 · Cited by 55
Jozaghi, E., Greer, A. M., Lampkin, H., & Buxton, J. A. (2018) · Substance Abuse Treatment Prevention and Policy · doi:10.1186/s13011-018-0158-1 · Cited by 54
Schitter, A. M., Fleckenstein, J., Frei, P., Taeymans, J., Kurpiers, N., & Radlinger, L. (2020) · PLoS ONE · doi:10.1371/journal.pone.0229705 · Cited by 53
Frandsen, T. F., Gildberg, F. A., & Tingleff, E. B. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.06.013 · Cited by 50
“Google Scholar is an example of a database with extensive coverage but severe limitations of the search interface (45).”
BackgroundDeveloping and Applying a Protocol for a Systematic Review in the Social Sciences
Campbell, A., Taylor, B. J., Bates, J. A., & O'Connor, Ú. (2017) · New Review of Academic Librarianship · doi:10.1080/13614533.2017.1281827 · Cited by 45
Ferrúa, C. P., Giorgi, R., Camerini, L., do Amaral, C. C., Ghisleni, G., Pinheiro, R. T., & Nedel, F. (2019) · Journal of Chemical Neuroanatomy · doi:10.1016/j.jchemneu.2019.101650 · Cited by 44
What do we study when we study misinformation? A scoping review of experimental research (2016-2022)
Murphy, G., Laurent, C. D. S., Reynolds, M., Aftab, O., Hegarty, K., Sun, Y., & Greene, C. M. (2023) · Harvard Kennedy School Misinformation Review · doi:10.37016/mr-2020-130 · Cited by 63
Bates, J. A., Best, P., McQuilkin, J., & Taylor, B. J. (2016) · The Journal of Academic Librarianship · doi:10.1016/j.acalib.2016.11.003 · Cited by 49
“, 2015), for example, Google Scholar will truncate search strings after 256 characters (Boeker et al., 2013).”
BackgroundSturm, B., & Sunyaev, A. (2018) · Business & Information Systems Engineering · doi:10.1007/s12599-018-0569-6 · Cited by 44
“…2013; Samadzadeh et al. 2013), academic web search engines are widely criticized for their minimalistic search interfaces, fluctuating and nontransparent search indices, low document quality, and export limitations (e.g., Asher et al. 2013; Boeker et al. 2013; Lewandowski 2010; Wu and Chen 2014).”
BackgroundInfluentialIrreproducibility in searches of scientific literature: A comparative analysis
Pozsgai, G., Löveï, G. L., Vasseur, L., Gurr, G. M., Batáry, P., Korponai, J. L., Littlewood, N. A., Liu, J., Móra, A., Obrycki, J. J., et al. (2021) · Ecology and Evolution · doi:10.1002/ece3.8154 · Cited by 45
“…used search platforms, Google Scholar, PubMed, Scopus, and Web of Science, are known to yield different results for the same search strings (Boeker et al., 2013; Gavel & Iselid, 2008; Gusenbauer & Haddaway, 2019 During a recent systematic review in ecology, we accidentally discovered that…”
Can Google Scholar and Mendeley help to assess the scholarly impacts of dissertations?
Kousha, K., & Thelwall, M. A. (2019) · Journal of Informetrics · doi:10.1016/j.joi.2019.02.009 · Cited by 41
Smart Home Devices for Supporting Older Adults: A Systematic Review
Ghafurian, M., Wang, K., Dhode, I., Kapoor, M., Morita, P. P., & Dautenhahn, K. (2023) · IEEE Access · doi:10.1109/access.2023.3266647 · Cited by 43
“Google Scholar was not used because of its limitations, such as limitations in the search functionalities and its precision [14].”
MethodHeck, T., Keller, C., & Rittberger, M. (2023) · International Journal on Digital Libraries · doi:10.1007/s00799-023-00364-3 · Cited by 57
Rodríguez-Molina, D., Mang, P., Schmitt, H., Chifiriuc, M. C., Radon, K., & Wengenroth, L. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1236-9 · Cited by 38
Silveira, R. G., Ferrúa, C. P., do Amaral, C. C., Garcia, T. F., de Souza, K. B., & Nedel, F. (2020) · Brain Research Bulletin · doi:10.1016/j.brainresbull.2020.01.009 · Cited by 37
Racism against healthcare users in inpatient care: a scoping review
Merz, S., Aksakal, T., Hibtay, A., Yücesoy, H., Fieselmann, J., Annaç, K., Yılmaz-Aslan, Y., Brzoska, P., & Tezcan‐Güntekin, H. (2024) · International Journal for Equity in Health · doi:10.1186/s12939-024-02156-w · Cited by 38
“While Google Scholar might have significant shortcomings for systematic reviews which limits its utility as a professional research tool [48, 49], it can be beneficial for the identification of grey literature [50].”
BackgroundFagan, J. C. (2017) · Information Technology and Libraries · doi:10.6017/ital.v36i2.9718 · Cited by 43
A Systematic Review of Emerging Ventilated Acoustic Metamaterials for Noise Control
Ang, L. Y. L., Cui, F., Lim, K. M., & Lee, H. P. (2023) · Sustainability · doi:10.3390/su15054113 · Cited by 34
Systematic Review of Social Robots for Health and Wellbeing: A Personal Healthcare Journey Lens
Ghafurian, M., Chandra, S., Hutchinson, R., Lim, A., Baliyan, I., Rhim, J., Gupta, G., Aroyo, A. M., Rasouli, S., & Dautenhahn, K. (2024) · ACM Transactions on Human-Robot Interaction · doi:10.1145/3700446 · Cited by 34
“We did not use Google Scholar because of its low precision and limited searching functionalities [23].”
Empirical Legal Research: A Guidance Book for Lawyers, Legislators and Regulators
Leeuw, F. L., & Schmeets, H. (2016) · Cited by 31
Differences between Altmetric Data Sources – A Case Study
Bar‐Ilan, J., Halevi, G., & Milojević, S. (2019) · Journal of Altmetrics · doi:10.29024/joa.4 · Cited by 30
Psychological flexibility in older adulthood: a scoping review.
Plys, E., Jacobs, M. L., Allen, R. S., & Arch, J. J. (2023) · Aging & Mental Health · doi:10.1080/13607863.2022.2036948 · Cited by 29
“Although some studies report that Google Scholar covered 100% of articles identified through gold standard databases (Gehanno et al., 2013), others reported that this search engine has less coverage and precision than traditional academic databases (Boeker et al., 2013; Bramer et al., 2016).”
BackgroundLunny, C., Salzwedel, D. M., Liu, T. W. B., Ramasubbu, C., Gerrish, S., Puil, L. J., Mintzes, B. J., & Wright, J. M. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.09.022 · Cited by 29
Katzenellenbogen, J. M., Woods, J. A., Teng, T. K., & Thompson, S. (2015) · BMC Cardiovascular Disorders · doi:10.1186/s12872-015-0081-6 · Cited by 28
“Although Google Scholar allows searching the full text of publications, thereby allowing recognition of data that may not be identified through subject headings keywords, titles and abstracts, this interface has less sophisticated functionality for systematic searching than traditional databases [79].”
BackgroundDemeulenaere, X. (2019) · Research in Transportation Economics · doi:10.1016/j.retrec.2019.100738 · Cited by 29
Specialist Bibliographic Databases
Gasparyan, A. Y., Yessirkepov, M., Voronov, A. A., Trukhachev, V. I., Kostyukova, E. I., Gerasimov, A. N., & Kitas, G. D. (2016) · Journal of Korean Medical Science · doi:10.3346/jkms.2016.31.5.660 · Cited by 33
“However, some experts are critical of the Google approach because of its low specificity for locating relevant primary sources (11) and its low functionality for structured systematic searches (12).”
BackgroundInformation retrieval in systematic reviews: a case study of the crime prevention literature
Tompson, L., & Belur, J. (2015) · Journal of Experimental Criminology · doi:10.1007/s11292-015-9243-x · Cited by 26
“from electronic databases, the limitations of its functionality mean that it has yet to usurp professional literature retrieval tools (Boeker et al. 2013).”
BackgroundJustesen, T. F., Freyberg, J., & Schultz, A. N. Ø. (2021) · BMC Medical Research Methodology · doi:10.1186/s12874-021-01281-2 · Cited by 34
“Despite the advantages of Google Scholar, its precision in structured literature searches has previously reported to be low [22, 23].”
BackgroundMethodApplicability of smart construction technology: Prioritization and future research directions
Ahn, H., Lee, C., Kim, M., Kim, T., Lee, D., Kwon, W., & Cho, H. (2023) · Automation in Construction · doi:10.1016/j.autcon.2023.104953 · Cited by 25
Mosha, N. F., & Ngulube, P. (2023) · Information · doi:10.3390/info14080427 · Cited by 40
“GS is a web search engine that may gather trustworthy material that occasionally may not be found in other academic or scientific databases [38].”
BackgroundÖrnek, Ö. K., Weinmann, T., Waibel, J., & Radon, K. (2020) · Systematic Reviews · doi:10.1186/s13643-020-01313-w · Cited by 25
“However, we address publication bias through funnel plots [41].”
MethodGender and health social enterprises in Africa: a research agenda
McKague, K., & Harrison, S. J. (2019) · International Journal for Equity in Health · doi:10.1186/s12939-019-0994-2 · Cited by 26
“Although a debate in the literature exists [3, 14], a recent study has found that Google Scholar is 100%”
BackgroundDeri, S., Rappaz, J., Aiello, L. M., & Quercia, D. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274312 · Cited by 23
“edinandreferencingby the seed papers. Independent from the initial seed set, we queried Google Scholar—which has been shown in prior studies to have comparable recall and precision to similar corpora [16, 18, 41, 96]—to get additional sources. We refined the search strategy iteratively to increase the recall of known…”
MethodNow is the time: VFR travel desperately seeking respect
Zentveld, E., Labas, A., Edwards, S., & Morrison, A. M. (2022) · International Journal of Tourism Research · doi:10.1002/jtr.2509 · Cited by 24
A literature review to identify factors that determine policies for influenza vaccination
Silva, M.-L., Lionel Perrier, A., Cohen, J. M., Paget, W., Mosnier, A., & Späth, H. M. (2015) · Health Policy · doi:10.1016/j.healthpol.2015.04.006 · Cited by 22
Stakeholders in the acquisition, preservation, and dissemination of indigenous knowledge projects
Chigwada, J. P., & Ngulube, P. (2023) · Information Development · doi:10.1177/02666669231192851 · Cited by 21
“These databases were chosen since they index peer-reviewed scholarly articles that are not predatory, and in order to accommodate articles that are not indexed in the three aggregators, Google Scholar was used to supplement the data (Boeker et al., 2013).”
Social Network Analysis in Software Development Projects: A Systematic Literature Review
Schreiber, R. R., & Zylka, M. P. (2020) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s021819402050014x · Cited by 20
Reviewing Automated Sensor-Based Visitor Tracking Studies: Beyond Traditional Observational Methods?
Mygind, L., & Bentsen, P. (2017) · Visitor Studies · doi:10.1080/10645578.2017.1404351 · Cited by 19
Delgado‐Quirós, L., Aguillo, I. F., Martín‐Martín, A., López‐Cózar, E. D., Orduña‐Malea, E., & Ortega, J. L. (2023) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24839 · Cited by 24
“These retrieval problems lead us to point out that limited search functionalities would influence on the matching of documents, distorting the real coverage, as could be the case of Google Scholar (Boeker et al., 2013; Gusenbauer & Haddaway, 2020).”
BackgroundKirk, N., & Cradock-Henry, N. A. (2022) · Land · doi:10.3390/land11060791 · Cited by 18
Information sources for obesity prevention policy research: a review of systematic reviews
Hanneke, R., & Young, S. K. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0543-2 · Cited by 19
Pulikowski, A., & Matysek, A. (2021) · The Journal of Academic Librarianship · doi:10.1016/j.acalib.2021.102417 · Cited by 19
Wohlgemut, J. M., Pisirir, E., Kyrimi, E., Stoner, R. S., Marsh, W., Perkins, Z., & Tai, N. R. M. (2023) · JAMIA Open · doi:10.1093/jamiaopen/ooad051 · Cited by 19
“The authors considered Google Scholar as a search engine in order to broaden the review’s inclusion, but decided against it due to evidence reporting its imprecision as a systematic search engine.(84,85) Fourth, studies were not excluded based on assessed quality, and 5 did not use validated methods to assess usabilit…”
Diversity and participation in private apartment buildings: a review of the literature
Liu, E., Easthope, H., Ho, C., & Buckle, C. (2018) · Geographical Research · doi:10.1111/1745-5871.12282 · Cited by 17
Lang, S., Tebben, E., Odean, R., Wells, M. B., & Huang, H. (2023) · Child & Youth Care Forum · doi:10.1007/s10566-023-09748-7 · Cited by 17
“…and orders results, we considered sixty consecutive irrelevant records to be an indicator that we had reached the end of relevant results as Google Scholar returns a greater volume of results than other search strategies, but these results tend to be less precise and relevant (Boeker et al., 2013).”
Useche, S. A., Faus, M., & Alonso, F. (2024) · Frontiers in Public Health · doi:10.3389/fpubh.2024.1335209 · Cited by 17
“Google Scholar was also excluded due to its overrepresentation of gray (non-peer-reviewed) literature (45) and concerns about its scientific precision, as noted in preceding literature (46).”
BackgroundGusenbauer, M. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1746 · Cited by 36
“Database analyses are focused on one or a few databases (e.g., Boeker et al. 1 ) or a narrow range of characteristics, such as subject coverage (e.g., Martín-Martín et al. 2 ).”
Following the Trail of eSports
Vera, J. A. C., Terrón, J. M. A., & Gómez‐García, S. (2018) · International Journal of Gaming and Computer-Mediated Simulations · doi:10.4018/ijgcms.2018100103 · Cited by 16
What is the student experience of remote proctoring? A pragmatic scoping review
Marano, E., Newton, P. M., Birch, Z., Croombs, M., Gilbert, C., & Draper, M. J. (2024) · Higher Education Quarterly · doi:10.1111/hequ.12506 · Cited by 27
Oliván, J. A. S., Cuenca, G. M., & Avilés, R. A. (2018) · Revista española de Documentación Científica · doi:10.3989/redc.2018.2.1491 · Cited by 20
Larsen, K. R. T., Hekler, E. B., Paul, M. J., & Gibson, B. (2020) · Communications of the Association for Information Systems · doi:10.17705/1cais.04601 · Cited by 14
Walters, W. H. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2503051122 · Cited by 14
Browse to search, visualize to explore: Who needs an alternative information retrieving model?
Pajić, D. (2014) · Computers in Human Behavior · doi:10.1016/j.chb.2014.07.010 · Cited by 14
Women green entrepreneurship: Conceptualisation and use of bricolage for policy recommendations
Potluri, S., Ananthram, S., & Phani, B. V. (2024) · Business Strategy and the Environment · doi:10.1002/bse.3764 · Cited by 13
Rock the Boat! Shaken by the COVID-19 Crisis: A Review on Teachers’ Competencies in ICT
Montiel, H., & Zermeño, M. G. G. (2022) · Frontiers in Education · doi:10.3389/feduc.2021.770442 · Cited by 18
“Although there is criticism for the use of Google Scholar as a source of scienti fi c information, as seen in Beel and Gipp (2010), Jacsó (2012), Mayr and Walter (2008), and Boeker et al. (2013), there is also evidence presented in Halevi et al. (2017) of its advantages over controlled databases.”
Metadata
- Authors
- Boeker, M., Vach, W., & Motschall, E.
- Affiliations
- University Medical Center Freiburg (DE)
- Corresponding author
- Boeker, Martin
- Venue
- BMC Medical Research Methodology
- Published
- 2013-10-26
- Volume
- 13
- Issue
- 1
- Pages
- 131-131
- ISSN
- 1471-2288 (ISSN-L)
- PMC
- PMC3840556
- OpenAlex
- W2073728848
- Cited by
- 225
- References
- 58
- Citations by year
- 2026: 17 · 2025: 22 · 2024: 27 · 2023: 22 · 2022: 22 · 2021: 10 · 2020: 16 · 2019: 19 · 2018: 16 · 2017: 16 · 2016: 15 · 2015: 13 · 2014: 8 · 2013: 1
- FWCI
- 13.59
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- BMC Medical Research Methodology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- FreiDok plus (Universitätsbibliothek Freiburg) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- Google Scholar, systematic search, search recall, search precision, search strategies, MEDLINE
- MeSH terms
- Abstracting and Indexing, Algorithms, Humans, Review Literature as Topic, MEDLINE, Internet, Data Mining, Search Engine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
