Google Scholar as replacement for systematic literature searches: good relative recall and precision are not enough
Open accessAbstract
BACKGROUND: Recent research indicates a high recall in Google Scholar searches for systematic reviews. These reports raised high expectations of Google Scholar as a unified and easy to use search interface. However, studies on the coverage of Google Scholar rarely used the search interface in a realistic approach but instead merely checked for the existence of gold standard references. In addition, the severe limitations of the Google Search interface must be taken into consideration when comparing with professional literature retrieval tools.The objectives of this work are to measure the relative recall and precision of searches with Google Scholar under conditions which are derived from structured search procedures conventional in scientific literature retrieval; and to provide an overview of current advantages and disadvantages of the Google Scholar search interface in scientific literature retrieval. METHODS: General and MEDLINE-specific search strategies were retrieved from 14 Cochrane systematic reviews. Cochrane systematic review search strategies were translated to Google Scholar search expression as good as possible under consideration of the original search semantics. The references of the included studies from the Cochrane reviews were checked for their inclusion in the result sets of the Google Scholar searches. Relative recall and precision were calculated. RESULTS: We investigated Cochrane reviews with a number of included references between 11 and 70 with a total of 396 references. The Google Scholar searches resulted in sets between 4,320 and 67,800 and a total of 291,190 hits. The relative recall of the Google Scholar searches had a minimum of 76.2% and a maximum of 100% (7 searches). The precision of the Google Scholar searches had a minimum of 0.05% and a maximum of 0.92%. The overall relative recall for all searches was 92.9%, the overall precision was 0.13%. CONCLUSION: The reported relative recall must be interpreted with care. It is a quality indicator of Google Scholar confined to an experimental setting which is unavailable in systematic retrieval due to the severe limitations of the Google Scholar search interface. Currently, Google Scholar does not provide necessary elements for systematic scientific literature retrieval such as tools for incremental query optimization, export of a large number of references, a visual search builder or a history function. Google Scholar is not ready as a professional searching tool for tasks where structured retrieval methodology is necessary.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The ability of a reviewer to manipulate their search query depends on the capabilities of a search system and thus they significantly influence recall and precision.
“The ability of a reviewer to manipulate their search query depends on the capabilities of a search system and thus they significantly influence recall and precision.25”
Quality criteria have previously been used to evaluate search systems in terms of recall and precision.
“Quality criteria have previously been used to evaluate search systems in terms of recall and precision indicating their suitability of search in general and systematic search in particular.”
This research fills a need for support in the choice of search system, currently lacking in evidence-synthesis methodology, and follows calls for comparative studies on the effectiveness of search systems or the 'need to develop 'bias profiles' for search engines.'
“This research thus fills a need for support in the choice of search system, currently lacking in evidence‐synthesis methodology, and follows calls for comparative studies on the effectiveness of search systems25 or the “need to develop ‘bias profiles’ for search engines.”( 35 , p1193)”
Google Scholar's search precision has been found to be significantly lower than 1% for systematic searches.
“Nevertheless, overall, Google Scholar's search precision has been found to be significantly lower than 1% for systematic searches.25”
Google Scholar's coverage and recall is an inadequate reason to use it as principal search system in systematic searches.
“Our findings support the criticism of Bramer et al,33 Bramer et al,34 and Boeker et al25 and indicate that Google Scholar's coverage and recall is an inadequate reason to use it as principal search system in systematic searches.53”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000100 · Cited by 28,164 · see citing works
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048 · see citing works
Cochrane Handbook for Systematic Reviews of Interventions
Tarsilla, M. (2010) · Journal of MultiDisciplinary Evaluation · doi:10.56645/jmde.v6i14.284 · Cited by 5,228 · see citing works
Comparison of PubMed, Scopus, Web of Science, and Google Scholar: strengths and weaknesses
Falagas, M. E., Pitsouni, E. I., Malietzis, G. A., & Pappas, G. (2008) · The FASEB Journal · doi:10.1096/fj.07-9492lsf · Cited by 5,039 · see citing works
A catalogue of reporting guidelines for health research
Simera, I., Moher, D., Hoey, J., Schulz, K. F., & Altman, D. G. (2010) · European Journal of Clinical Investigation · doi:10.1111/j.1365-2362.2009.02234.x · Cited by 1,047 · see citing works
Kulkarni, A. V. (2009) · JAMA · doi:10.1001/jama.2009.1307 · Cited by 821 · see citing works
Detection of gene pathways with predictive power for breast cancer prognosis
Ma, S., & Kosorok, M. R. (2010) · BMC Bioinformatics · doi:10.1186/1471-2105-11-1 · Cited by 691 · see citing works
An evidence-based practice guideline for the peer review of electronic search strategies
Sampson, M., McGowan, J., Cogo, E., Grimshaw, J., Moher, D., & Lefebvre, C. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2008.10.012 · Cited by 608 · see citing works
Lefebvre, C., Manheimer, E. W., & Glanville, J. (2008) · doi:10.1002/9780470712184.ch6 · Cited by 599 · see citing works
Selecting Studies and Collecting Data
Higgins, J. P., & Deeks, J. J. (2008) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9780470712184.ch7 · Cited by 467 · see citing works
Is the coverage of google scholar enough to be used alone for systematic reviews
Gehanno, J.-F., Rollin, L., & Darmoni, S. (2013) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-13-7 · Cited by 357 · see citing works
Antidepressants versus placebo for depression in primary care
Arroll, B., Elley, C. R., Fishman, T., Goodyear‐Smith, F., Kenealy, T. W., Blashki, G. A., Kerse, N. M., & MacGillivray, S. A. (2009) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd007954 · Cited by 353 · see citing works
Semi-automated screening of biomedical citations for systematic reviews
Wallace, B. C., Trikalinos, T. A., Lau, J., Brodley, C., & Schmid, C. H. (2010) · BMC Bioinformatics · doi:10.1186/1471-2105-11-55 · Cited by 351 · see citing works
Green tea (Camellia sinensis) for the prevention of cancer
Filippini, T., Malavolti, M., Borrelli, F., Izzo, A. A., Fairweather‐Tait, S. J., Horneber, M., & Vinceti, M. (2020) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd005004.pub3 · Cited by 328 · see citing works
Non-pharmacological interventions for assisting the induction of anaesthesia in children
Manyande, A., Cyna, A. M., Yip, P., Chooi, C., & Middleton, P. (2015) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd006447.pub3 · Cited by 324 · see citing works
Artemisinin-based combination therapy for treating uncomplicated malaria
Sinclair, D., Zani, B., Donegan, S., Olliaro, P., & Garner, P. (2009) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd007483.pub2 · Cited by 289 · see citing works
Brief interventions for heavy alcohol users admitted to general hospital wards
McQueen, J., Howe, T. E., Allan, L., Mains, D., & Hardy, V. (2011) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd005191.pub3 · Cited by 284 · see citing works
"Brimful of STARLITE": toward standards for reporting literature searches.
Booth, A. (2006) · Europe PMC (PubMed Central) · Cited by 267 · see citing works
Systematic reviews need systematic searchers.
(2005) · Europe PMC (PubMed Central) · Cited by 257 · see citing works
do Nascimento, P., Módolo, N. S. P., Andrade, S., Guimarães, M. M., Braz, L. G., & Dib, R. E. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd006058.pub3 · Cited by 240 · see citing works
Comparing test searches in PubMed and Google Scholar
Shultz, M. (2007) · Journal of the Medical Library Association JMLA · doi:10.3163/1536-5050.95.4.442 · Cited by 231 · see citing works
Oral iron supplements for children in malaria-endemic areas
Neuberger, A., Okebe, J., Yahav, D., & Paul, M. (2016) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd006589.pub4 · Cited by 220 · see citing works
Interventions to prevent occupational noise-induced hearing loss
Tikka, C., Verbeek, J. H. A. M., Kateman, E., Morata, T. C., Dreschler, W. A., & Ferrite, S. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd006396.pub4 · Cited by 213 · see citing works
Unpacking your literature search toolbox: on search styles and tactics
Booth, A. (2008) · Health Information & Libraries Journal · doi:10.1111/j.1471-1842.2008.00825.x · Cited by 193 · see citing works
Google Scholar is not enough to be used alone for systematic reviews
Giustini, D., & Kamel Boulos, M. N. (2013) · Online Journal of Public Health Informatics · doi:10.5210/ojphi.v5i2.4623 · Cited by 187 · see citing works
Green tea (Camellia sinensis) for the prevention of cancer
Boehm, K., Borrelli, F., Ernst, E., Habacher, G., Hung, S. K., Milazzo, S., & Horneber, M. (2009) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd005004.pub2 · Cited by 180 · see citing works
Therapeutic interventions for symptomatic treatment in Huntington's disease
Mestre, T., Ferreira, J. J., Coelho, M. M., Rosa, M. M., & Sampaio, C. (2009) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd006456.pub2 · Cited by 173 · see citing works
The effects of antimicrobial therapy on bacterial vaginosis in non-pregnant women
Oduyebo, O. O., Anorlu, R. I., & Ogunsola, F. T. (2009) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd006055.pub2 · Cited by 166 · see citing works
Haig, A., & Dozier, M. (2003) · Medical Teacher · doi:10.1080/01421590310001608667 · Cited by 163 · see citing works
Pharmacotherapy for anxiety disorders in children and adolescents
Ipser, J., Stein, D. J., Hawkridge, S. M., & Hoppe, L. J. (2009) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd005170.pub2 · Cited by 160 · see citing works
Bohlius, J., Schmidlin, K., Brillant, C., Schwarzer, G., Trelle, S., Seidenfeld, J., Zwahlen, M., Clarke, M. J., Weingart, O., Kluge, S., et al. (2009) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd007303.pub2 · Cited by 151 · see citing works
Retrieving Clinical Evidence: A Comparison of PubMed and Google Scholar for Quick Clinical Searches
Shariff, S. Z., Bejaimal, S. A. D., Sontrop, J. M., Iansavichus, A. V., Haynes, R. B., Weir, M. A., & Garg, A. X. (2013) · Journal of Medical Internet Research · doi:10.2196/jmir.2624 · Cited by 148 · see citing works
Errors in search strategies were identified by type and frequency
Sampson, M., & McGowan, J. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.01.007 · Cited by 142 · see citing works
Sampson, M., Zhang, L., Morrison, A., Barrowman, N. J., Clifford, T., Platt, R. W., Klassen, T. P., & Moher, D. (2006) · BMC Medical Research Methodology · doi:10.1186/1471-2288-6-33 · Cited by 133 · see citing works
Comparison of PubMed and Google Scholar Literature Searches
Anders, M., & Evans, D. P. (2010) · Respiratory Care · doi:10.4187/respcare.10550578 · Cited by 117 · see citing works
Medical literature searches: a comparison of P ub M ed and G oogle S cholar
Nourbakhsh, E., Nugent, R., Wang, H., Çevik, C., & NUGENT, K. M. (2012) · Health Information & Libraries Journal · doi:10.1111/j.1471-1842.2012.00992.x · Cited by 113 · see citing works
An exploratory study of Google Scholar
Mayr, P., & Walter, A. (2007) · Online Information Review · doi:10.1108/14684520710841784 · Cited by 101 · see citing works
García‐Pérez, M. A. (2010) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.21372 · Cited by 101 · see citing works
Non-pharmacological interventions for assisting the induction of anaesthesia in children
Yip, P., Middleton, P., Cyna, A. M., & Carlyle, A. V. (2009) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd006447.pub2 · Cited by 96 · see citing works
Google Scholar Search Performance: Comparative Recall and Precision
Walters, W. H. (2009) · portal Libraries and the Academy · doi:10.1353/pla.0.0034 · Cited by 93 · see citing works
Guimarães, M. M., Dib, R. E., Smith, A. F., & Matos, D. (2009) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd006058.pub2 · Cited by 93 · see citing works
How much searching is enough? Comprehensive versus optimal retrieval for technology assessments
Booth, A. (2010) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462310000966 · Cited by 88 · see citing works
Oral iron supplements for children in malaria-endemic areas
Okebe, J., Yahav, D., Shbita, R., & Paul, M. (2011) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd006589.pub3 · Cited by 88 · see citing works
Precision of healthcare systematic review searches in a cross‐sectional sample
Sampson, M., Tetzlaff, J., & Urquhart, C. (2011) · Research Synthesis Methods · doi:10.1002/jrsm.42 · Cited by 80 · see citing works
Reproducibility of Literature Search Reporting in Medical Education Reviews
Maggio, L. A., Tannery, N. H., & Kanter, S. L. (2011) · Academic Medicine · doi:10.1097/acm.0b013e31822221e7 · Cited by 77 · see citing works
Bar‐On, E. S., Goldberg, E., Hellmann, S., & Leibovici, L. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd005530.pub3 · Cited by 70 · see citing works
Interventions to prevent occupational noise-induced hearing loss
Verbeek, J. H. A. M., Kateman, E., Morata, T. C., Dreschler, W. A., & Tikka, C. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd006396.pub3 · Cited by 69 · see citing works
A look at Google Scholar, PubMed, and Scirus: comparisons and recommendations
Giustini, D., & Barsky, E. (2005) · Journal of the Canadian Health Libraries Association / Journal de l Association de bilbiothèques de la santé du Canada · doi:10.5596/c05-030 · Cited by 60 · see citing works
Guimarães, M. M., Dib, R. E., Smith, A. F., Matos, D., & Gomes, B. P. F. D. A. (2009) · Anesthesia & Analgesia · doi:10.1213/ane.0b013e3181bb1a36 · Cited by 49 · see citing works
Waffenschmidt, S., Janzen, T., Hausner, E., & Kaiser, T. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.11.011 · Cited by 40 · see citing works
Haase, A., Follmann, M., Skipka, G., & Kirchner, H. (2007) · BMC Medical Research Methodology · doi:10.1186/1471-2288-7-28 · Cited by 36 · see citing works
Single dose oral flurbiprofen for acute postoperative pain in adults
Sultan, A., McQuay, H. J., Moore, R. A., & Derry, S. (2009) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd007358.pub2 · Cited by 33 · see citing works
Lost in PubMed. Factors influencing the success of medical information retrieval
Vanopstal, K., Buysschaert, J., Laureys, G., & Stichele, R. V. (2013) · Expert Systems with Applications · doi:10.1016/j.eswa.2013.01.036 · Cited by 17 · see citing works
Analysis of equine protein-coding gene structure and expression by RNA-sequencing
Coleman, S. J., Zeng, Z., Liu, J., & MacLeod, J. N. (2010) · BMC Bioinformatics · doi:10.1186/1471-2105-11-s4-o8 · Cited by 11 · see citing works
Boeker, M., Vach, W., & Motschall, E. (2012) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.01.015 · Cited by 9 · see citing works
Self-citation260 Interventions to prevent occupational noise-induced hearing loss - a Cochrane Review
Verbeek, V. J., Kateman, Morata, Dreschler, & Mischke (2013) · Occupational and Environmental Medicine · doi:10.1136/oemed-2013-101717.260 · Cited by 6 · see citing works
Reflections on medical literature searches
Sharma, P. (2013) · Transfusion · doi:10.1111/trf.12247 · Cited by 1 · see citing works
Flagged first, then by citations
1 of the 58 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- Ovaska T Fiander M Weightman A BMC medical informatics and decision making – user comments – is the coverage of Google scholar enough to be used alone for systematic reviews 2013 http://www.biomedcentral.com/1472-6947/13/7/comments
Cited by
Sorted by citations
The role of google scholar in evidence reviews and its applicability to grey literature searching
Haddaway, N. R., Collins, A. M., Coughlin, D., & Kirk, S. (2015) · PLOS ONE · doi:10.1371/journal.pone.0138237 · Cited by 2,387 · see citing works
Gusenbauer, M., & Haddaway, N. R. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1378 · Cited by 2,379 · see citing works
“25 This is not surprising, since our findings show that Google Scholar does not support many of the features required for systematic searches.”
BackgroundResultInfluentialHalevi, G., Moed, H., & Bar-Ilan, J. (2017) · Journal of Informetrics · doi:10.1016/j.joi.2017.06.005 · Cited by 488 · see citing works
Consolidated principles for screening based on a systematic review and consensus process
Dobrow, M. J., Hagens, V., Chafe, R., Sullivan, T., & Rabeneck, L. (2018) · Canadian Medical Association Journal · doi:10.1503/cmaj.171154 · Cited by 359 · see citing works
Wearable sensor-based real-time gait detection: A systematic review
Prasanth, H., Caban, M., Keller, U., Courtine, G., Ijspeert, A., Vallery, H., & von Zitzewitz, J. (2021) · Sensors · doi:10.3390/s21082727 · Cited by 348 · see citing works
Newton, P. M. (2018) · Frontiers in Education · doi:10.3389/feduc.2018.00067 · Cited by 289 · see citing works
Optimal literature search for systematic reviews in surgery
Gooßen, K., Tenckhoff, S., Probst, P., Grummich, K., Mihaljević, A. L., Büchler, M. W., & Diener, M. K. (2017) · Langenbeck s Archives of Surgery · doi:10.1007/s00423-017-1646-x · Cited by 247 · see citing works
Robotic technologies for on-site building construction: A systematic review
Gharbia, M., Chang-Richards, A. Y., Lu, Y., Zhong, R. Y., & Li, H. (2020) · Journal of Building Engineering · doi:10.1016/j.jobe.2020.101584 · Cited by 233 · see citing works
Powered exoskeletons for bipedal locomotion after spinal cord injury
Contreras-Vidal, J. L., Bhagat, N. A., Brantley, J. A., Cruz-Garza, J. G., He, Y., Manley, Q., Nakagome, S., Nathan, K., Tan, S. H., Zhu, F., et al. (2016) · Journal of Neural Engineering · doi:10.1088/1741-2560/13/3/031001 · Cited by 217 · see citing works
Bramer, W. M., Giustini, D., & Kramer, B. M. R. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0215-7 · Cited by 202 · see citing works
“Low precision has always been considered a problem in GS [11], but when accounting for actually usable search results (i.”
BackgroundOloyede, A. A., Faruk, N., Noma, N., Tebepah, E., & Nwaulune, A. K. (2023) · Heliyon · doi:10.1016/j.heliyon.2023.e17654 · Cited by 151 · see citing works
Newton, P. M., & Essex, K. (2023) · Journal of Academic Ethics · doi:10.1007/s10805-023-09485-5 · Cited by 175 · see citing works
Bramer, W. M., Giustini, D., Kramer, B. M., & Anderson, P. (2013) · Systematic Reviews · doi:10.1186/2046-4053-2-115 · Cited by 139 · see citing works
Main gamification concepts: A systematic mapping study
Rodrigues, L. F., Oliveira, A., & Rodrigues, H. (2019) · Heliyon · doi:10.1016/j.heliyon.2019.e01993 · Cited by 154 · see citing works
Modeling languages in Industry 4.0: an extended systematic mapping study
Wortmann, A., Barais, O., Combemale, B., & Wimmer, M. (2019) · Software & Systems Modeling · doi:10.1007/s10270-019-00757-6 · Cited by 136 · see citing works
Narcissism and Social Networking Behavior: A Meta‐Analysis
Gnambs, T., & Appel, M. (2017) · Journal of Personality · doi:10.1111/jopy.12305 · Cited by 135 · see citing works
A rapid method to increase transparency and efficiency in web-based searches
Haddaway, N. R., Collins, A. M., Coughlin, D., & Kirk, S. (2017) · Environmental Evidence · doi:10.1186/s13750-016-0079-2 · Cited by 129 · see citing works
Stansfield, C., Dickson, K., & Bangpan, M. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0371-9 · Cited by 122 · see citing works
A Cross-Domain Systematic Mapping Study on Software Engineering for Digital Twins
Dalibor, M., Jansen, N., Rumpe⋆, B., Schmalzing, D., Wachtmeister, L., Wimmer, M., & Wortmann, A. (2022) · Journal of Systems and Software · doi:10.1016/j.jss.2022.111361 · Cited by 126 · see citing works
Newton, P. M., & Salvi, A. (2020) · Frontiers in Education · doi:10.3389/feduc.2020.602451 · Cited by 118 · see citing works
Multi-Criteria Decision-Making Methods to Address Water Allocation Problems: A Systematic Review
Gebre, S. L., Cattrysse, D., & Orshoven, J. V. (2021) · Water · doi:10.3390/w13020125 · Cited by 114 · see citing works
Gnambs, T., & Kaspar, K. (2016) · Assessment · doi:10.1177/1073191115624547 · Cited by 106 · see citing works
Góngora, P., Rocks, S., Fahr, P., Rivero‐Arias, O., & Tsiachristas, A. (2022) · Value in Health · doi:10.1016/j.jval.2022.11.007 · Cited by 98 · see citing works
Crime scripting: A systematic review
Dehghanniri, H., & Borrion, H. (2019) · European Journal of Criminology · doi:10.1177/1477370819850943 · Cited by 102 · see citing works
Gebre, S. L., Cattrysse, D., Alemayehu, E., & Orshoven, J. V. (2021) · International Soil and Water Conservation Research · doi:10.1016/j.iswcr.2021.04.005 · Cited by 109 · see citing works
O'Connor, A. M. C., Anderson, K. M., Goodell, C. K., & Sargeant, J. M. (2014) · Zoonoses and Public Health · doi:10.1111/zph.12125 · Cited by 92 · see citing works
O'Connor, S. R., Connaghan, J., Maguire, R., Kotronoulas, G., Flannagan, C., Jain, S., Brady, N., & McCaughan, E. (2018) · Patient Education and Counseling · doi:10.1016/j.pec.2018.12.015 · Cited by 88 · see citing works
Model composition in Model Driven Engineering: A systematic literature review
Abouzahra, A., Sabraoui, A., & Afdel, K. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106316 · Cited by 87 · see citing works
Predatory publications in evidence syntheses
Ross‐White, A., Godfrey, C. M., Sears, K., & Wilson, R. A. (2019) · Journal of the Medical Library Association JMLA · doi:10.5195/jmla.2019.491 · Cited by 82 · see citing works
A systematic scoping review of the Social Vulnerability Index as applied to natural hazards
Painter, M. A., Shah, S. H., Damestoit, G. C., Khalid, F., Prudencio, W., Chisty, M. A., Tormos‐Aponte, F., & Wilhelmi, O. V. (2024) · Natural Hazards · doi:10.1007/s11069-023-06378-z · Cited by 90 · see citing works
Gusenbauer, M., & Haddaway, N. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1457 · Cited by 88 · see citing works
The Role of AI in Serious Games and Gamification for Health: Scoping Review
Tolks, D., Schmidt, J., & Kühn, S. (2023) · JMIR Serious Games · doi:10.2196/48258 · Cited by 82 · see citing works
Yuan, Z., Wen, B., He, C., Zhou, J., Zhou, Z., & Xu, F. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph19116572 · Cited by 83 · see citing works
Campo, K. X., Teper, T., Eaton, C., Shipman, A. M., Bhatia, G., & Mesmer, B. L. (2022) · Systems Engineering · doi:10.1002/sys.21644 · Cited by 76 · see citing works
TheoryOn: A Design Framework and System for Unlocking Behavioral Knowledge Through Ontology Learning
Li, J., Larsen, K. R. T., & Abbasi, A. (2020) · MIS Quarterly · doi:10.25300/misq/2020/15323 · Cited by 68 · see citing works
Steinau, S., Marrella, A., Andrews, K., Leotta, F., Mecella, M., & Reichert, M. (2018) · Software & Systems Modeling · doi:10.1007/s10270-018-0695-0 · Cited by 67 · see citing works
Christ, A. A., Capon-Sieber, V., Grob, U., & Praetorius, A.-K. (2022) · Studies in Educational Evaluation · doi:10.1016/j.stueduc.2022.101209 · Cited by 66 · see citing works
Linder, S. K., Kamath, G. R., Pratt, G. F., Saraykar, S. S., & Volk, R. J. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.10.008 · Cited by 63 · see citing works
Briscoe, S. (2015) · BMC Research Notes · doi:10.1186/s13104-015-1079-y · Cited by 57 · see citing works
Toward a Formalization of the Process to Select IMIA Yearbook Best Papers
Brigitte Séroussi, A., Griffon, N., Kerdelhué, G., Jaulent, M., Jacques Bouaud, A., & Lamy, J. B. (2014) · Methods of Information in Medicine · doi:10.3414/me14-01-0031 · Cited by 61 · see citing works
The Implementation of Internet Interventions for Depression: A Scoping Review
Drozd, F., Vaskinn, L., Bergsund, H. B., Haga, S. M., Slinning, K., & Bjørkli, C. A. (2016) · Journal of Medical Internet Research · doi:10.2196/jmir.5670 · Cited by 60 · see citing works
Larsen, K. R. T., Hovorka, D. S., Dennis, A. R., & West, J. D. (2019) · Journal of the Association for Information Systems · doi:10.17705/1jais.00556 · Cited by 64 · see citing works
Korać, S., Saliterer, I., & Weigand, B. (2018) · International Public Management Journal · doi:10.1080/10967494.2018.1430086 · Cited by 60 · see citing works
Hartmann, D., Wang, S. M., Pohlmann, L., & Berendt, B. (2025) · Journal of Computational Social Science · doi:10.1007/s42001-025-00381-z · Cited by 59 · see citing works
Last-Mile Strategies for Urban Freight Delivery: A Systematic Review
Lyons, T., & McDonald, N. (2022) · Transportation Research Record Journal of the Transportation Research Board · doi:10.1177/03611981221103596 · Cited by 55 · see citing works
Eysenbach, J. T. G. (2001) · Medical Informatics and the Internet in Medicine · doi:10.1080/14639230110075459 · Cited by 53 · see citing works
Jozaghi, E., Greer, A. M., Lampkin, H., & Buxton, J. A. (2018) · Substance Abuse Treatment Prevention and Policy · doi:10.1186/s13011-018-0158-1 · Cited by 49 · see citing works
Schitter, A. M., Fleckenstein, J., Frei, P., Taeymans, J., Kurpiers, N., & Radlinger, L. (2020) · PLoS ONE · doi:10.1371/journal.pone.0229705 · Cited by 46 · see citing works
Frandsen, T. F., Gildberg, F. A., & Tingleff, E. B. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.06.013 · Cited by 45 · see citing works
Developing and Applying a Protocol for a Systematic Review in the Social Sciences
Campbell, A., Taylor, B. J., Bates, J. A., & O'Connor, Ú. (2017) · New Review of Academic Librarianship · doi:10.1080/13614533.2017.1281827 · Cited by 45 · see citing works
Ferrúa, C. P., Giorgi, R., Camerini, L., do Amaral, C. C., Ghisleni, G., Pinheiro, R. T., & Nedel, F. (2019) · Journal of Chemical Neuroanatomy · doi:10.1016/j.jchemneu.2019.101650 · Cited by 44 · see citing works
What do we study when we study misinformation? A scoping review of experimental research (2016-2022)
Murphy, G., Laurent, C. D. S., Reynolds, M., Aftab, O., Hegarty, K., Sun, Y., & Greene, C. M. (2023) · Harvard Kennedy School Misinformation Review · doi:10.37016/mr-2020-130 · Cited by 44 · see citing works
Bates, J. A., Best, P., McQuilkin, J., & Taylor, B. J. (2016) · The Journal of Academic Librarianship · doi:10.1016/j.acalib.2016.11.003 · Cited by 44 · see citing works
Sturm, B., & Sunyaev, A. (2018) · Business & Information Systems Engineering · doi:10.1007/s12599-018-0569-6 · Cited by 44 · see citing works
Irreproducibility in searches of scientific literature: A comparative analysis
Pozsgai, G., Löveï, G. L., Vasseur, L., Gurr, G. M., Batáry, P., Korponai, J. L., Littlewood, N. A., Liu, J., Móra, A., Obrycki, J. J., et al. (2021) · Ecology and Evolution · doi:10.1002/ece3.8154 · Cited by 42 · see citing works
Can Google Scholar and Mendeley help to assess the scholarly impacts of dissertations?
Kousha, K., & Thelwall, M. A. (2019) · Journal of Informetrics · doi:10.1016/j.joi.2019.02.009 · Cited by 41 · see citing works
Smart Home Devices for Supporting Older Adults: A Systematic Review
Ghafurian, M., Wang, K., Dhode, I., Kapoor, M., Morita, P. P., & Dautenhahn, K. (2023) · IEEE Access · doi:10.1109/access.2023.3266647 · Cited by 40 · see citing works
Heck, T., Keller, C., & Rittberger, M. (2023) · International Journal on Digital Libraries · doi:10.1007/s00799-023-00364-3 · Cited by 38 · see citing works
Rodríguez-Molina, D., Mang, P., Schmitt, H., Chifiriuc, M. C., Radon, K., & Wengenroth, L. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1236-9 · Cited by 38 · see citing works
Silveira, R. G., Ferrúa, C. P., do Amaral, C. C., Garcia, T. F., de Souza, K. B., & Nedel, F. (2020) · Brain Research Bulletin · doi:10.1016/j.brainresbull.2020.01.009 · Cited by 37 · see citing works
Racism against healthcare users in inpatient care: a scoping review
Merz, S., Aksakal, T., Hibtay, A., Yücesoy, H., Fieselmann, J., Annaç, K., Yılmaz-Aslan, Y., Brzoska, P., & Tezcan‐Güntekin, H. (2024) · International Journal for Equity in Health · doi:10.1186/s12939-024-02156-w · Cited by 36 · see citing works
Fagan, J. C. (2017) · Information Technology and Libraries · doi:10.6017/ital.v36i2.9718 · Cited by 36 · see citing works
A Systematic Review of Emerging Ventilated Acoustic Metamaterials for Noise Control
Ang, L. Y. L., Cui, F., Lim, K. M., & Lee, H. P. (2023) · Sustainability · doi:10.3390/su15054113 · Cited by 34 · see citing works
Systematic Review of Social Robots for Health and Wellbeing: A Personal Healthcare Journey Lens
Ghafurian, M., Chandra, S., Hutchinson, R., Lim, A., Baliyan, I., Rhim, J., Gupta, G., Aroyo, A. M., Rasouli, S., & Dautenhahn, K. (2024) · ACM Transactions on Human-Robot Interaction · doi:10.1145/3700446 · Cited by 34 · see citing works
Empirical Legal Research: A Guidance Book for Lawyers, Legislators and Regulators
Leeuw, F. L., & Schmeets, H. (2016) · Cited by 31 · see citing works
Differences between Altmetric Data Sources – A Case Study
Bar‐Ilan, J., Halevi, G., & Milojević, S. (2019) · Journal of Altmetrics · doi:10.29024/joa.4 · Cited by 30 · see citing works
Psychological flexibility in older adulthood: a scoping review.
Plys, E., Jacobs, M. L., Allen, R. S., & Arch, J. J. (2023) · Aging & Mental Health · doi:10.1080/13607863.2022.2036948 · Cited by 29 · see citing works
“Although some studies report that Google Scholar covered 100% of articles identified through gold standard databases (Gehanno et al., 2013), others reported that this search engine has less coverage and precision than traditional academic databases (Boeker et al., 2013; Bramer et al., 2016).”
BackgroundLunny, C., Salzwedel, D. M., Liu, T. W. B., Ramasubbu, C., Gerrish, S., Puil, L. J., Mintzes, B. J., & Wright, J. M. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.09.022 · Cited by 29 · see citing works
Katzenellenbogen, J. M., Woods, J. A., Teng, T. K., & Thompson, S. (2015) · BMC Cardiovascular Disorders · doi:10.1186/s12872-015-0081-6 · Cited by 28 · see citing works
Demeulenaere, X. (2019) · Research in Transportation Economics · doi:10.1016/j.retrec.2019.100738 · Cited by 27 · see citing works
Specialist Bibliographic Databases
Gasparyan, A. Y., Yessirkepov, M., Voronov, A. A., Trukhachev, V. I., Kostyukova, E. I., Gerasimov, A. N., & Kitas, G. D. (2016) · Journal of Korean Medical Science · doi:10.3346/jkms.2016.31.5.660 · Cited by 27 · see citing works
Information retrieval in systematic reviews: a case study of the crime prevention literature
Tompson, L., & Belur, J. (2015) · Journal of Experimental Criminology · doi:10.1007/s11292-015-9243-x · Cited by 26 · see citing works
Justesen, T. F., Freyberg, J., & Schultz, A. N. Ø. (2021) · BMC Medical Research Methodology · doi:10.1186/s12874-021-01281-2 · Cited by 26 · see citing works
Applicability of smart construction technology: Prioritization and future research directions
Ahn, H., Lee, C., Kim, M., Kim, T., Lee, D., Kwon, W., & Cho, H. (2023) · Automation in Construction · doi:10.1016/j.autcon.2023.104953 · Cited by 25 · see citing works
Mosha, N. F., & Ngulube, P. (2023) · Information · doi:10.3390/info14080427 · Cited by 24 · see citing works
Örnek, Ö. K., Weinmann, T., Waibel, J., & Radon, K. (2020) · Systematic Reviews · doi:10.1186/s13643-020-01313-w · Cited by 25 · see citing works
Gender and health social enterprises in Africa: a research agenda
McKague, K., & Harrison, S. J. (2019) · International Journal for Equity in Health · doi:10.1186/s12939-019-0994-2 · Cited by 24 · see citing works
Deri, S., Rappaz, J., Aiello, L. M., & Quercia, D. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274312 · Cited by 23 · see citing works
Now is the time: VFR travel desperately seeking respect
Zentveld, E., Labas, A., Edwards, S., & Morrison, A. M. (2022) · International Journal of Tourism Research · doi:10.1002/jtr.2509 · Cited by 24 · see citing works
A literature review to identify factors that determine policies for influenza vaccination
Silva, M.-L., Lionel Perrier, A., Cohen, J. M., Paget, W., Mosnier, A., & Späth, H. M. (2015) · Health Policy · doi:10.1016/j.healthpol.2015.04.006 · Cited by 22 · see citing works
Stakeholders in the acquisition, preservation, and dissemination of indigenous knowledge projects
Chigwada, J. P., & Ngulube, P. (2023) · Information Development · doi:10.1177/02666669231192851 · Cited by 21 · see citing works
Social Network Analysis in Software Development Projects: A Systematic Literature Review
Schreiber, R. R., & Zylka, M. P. (2020) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s021819402050014x · Cited by 20 · see citing works
Reviewing Automated Sensor-Based Visitor Tracking Studies: Beyond Traditional Observational Methods?
Mygind, L., & Bentsen, P. (2017) · Visitor Studies · doi:10.1080/10645578.2017.1404351 · Cited by 19 · see citing works
Delgado‐Quirós, L., Aguillo, I. F., Martín‐Martín, A., López‐Cózar, E. D., Orduña‐Malea, E., & Ortega, J. L. (2023) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24839 · Cited by 19 · see citing works
Kirk, N., & Cradock-Henry, N. A. (2022) · Land · doi:10.3390/land11060791 · Cited by 18 · see citing works
Information sources for obesity prevention policy research: a review of systematic reviews
Hanneke, R., & Young, S. K. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0543-2 · Cited by 18 · see citing works
Pulikowski, A., & Matysek, A. (2021) · The Journal of Academic Librarianship · doi:10.1016/j.acalib.2021.102417 · Cited by 19 · see citing works
Wohlgemut, J. M., Pisirir, E., Kyrimi, E., Stoner, R. S., Marsh, W., Perkins, Z., & Tai, N. R. M. (2023) · JAMIA Open · doi:10.1093/jamiaopen/ooad051 · Cited by 19 · see citing works
Diversity and participation in private apartment buildings: a review of the literature
Liu, E., Easthope, H., Ho, C., & Buckle, C. (2018) · Geographical Research · doi:10.1111/1745-5871.12282 · Cited by 17 · see citing works
Lang, S., Tebben, E., Odean, R., Wells, M. B., & Huang, H. (2023) · Child & Youth Care Forum · doi:10.1007/s10566-023-09748-7 · Cited by 16 · see citing works
Useche, S. A., Faus, M., & Alonso, F. (2024) · Frontiers in Public Health · doi:10.3389/fpubh.2024.1335209 · Cited by 16 · see citing works
Gusenbauer, M. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1746 · Cited by 16 · see citing works
Following the Trail of eSports
Vera, J. A. C., Terrón, J. M. A., & Gómez‐García, S. (2018) · International Journal of Gaming and Computer-Mediated Simulations · doi:10.4018/ijgcms.2018100103 · Cited by 16 · see citing works
What is the student experience of remote proctoring? A pragmatic scoping review
Marano, E., Newton, P. M., Birch, Z., Croombs, M., Gilbert, C., & Draper, M. J. (2024) · Higher Education Quarterly · doi:10.1111/hequ.12506 · Cited by 27 · see citing works
Oliván, J. A. S., Cuenca, G. M., & Avilés, R. A. (2018) · Revista española de Documentación Científica · doi:10.3989/redc.2018.2.1491 · Cited by 15 · see citing works
Larsen, K. R. T., Hekler, E. B., Paul, M. J., & Gibson, B. (2020) · Communications of the Association for Information Systems · doi:10.17705/1cais.04601 · Cited by 14 · see citing works
Walters, W. H. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2503051122 · Cited by 14 · see citing works
Browse to search, visualize to explore: Who needs an alternative information retrieving model?
Pajić, D. (2014) · Computers in Human Behavior · doi:10.1016/j.chb.2014.07.010 · Cited by 14 · see citing works
Women green entrepreneurship: Conceptualisation and use of bricolage for policy recommendations
Potluri, S., Ananthram, S., & Phani, B. V. (2024) · Business Strategy and the Environment · doi:10.1002/bse.3764 · Cited by 13 · see citing works
Rock the Boat! Shaken by the COVID-19 Crisis: A Review on Teachers’ Competencies in ICT
Montiel, H., & Zermeño, M. G. G. (2022) · Frontiers in Education · doi:10.3389/feduc.2021.770442 · Cited by 13 · see citing works
- Authors
- Boeker, M., Vach, W., & Motschall, E.
- Affiliations
- University Medical Center Freiburg (DE)
- Corresponding author
- Boeker, Martin
- Venue
- BMC Medical Research Methodology
- Published
- 2013-10-26
- Volume
- 13
- Issue
- 1
- Pages
- 131-131
- ISSN
- 1471-2288 (ISSN-L)
- PMC
- PMC3840556
- OpenAlex
- W2073728848
- Cited by
- 225
- References
- 58
- Citations by year
- 2026: 17 · 2025: 22 · 2024: 27 · 2023: 22 · 2022: 22 · 2021: 10 · 2020: 16 · 2019: 19 · 2018: 16 · 2017: 16 · 2016: 15 · 2015: 13 · 2014: 8 · 2013: 1
- FWCI
- 13.59
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- BMC Medical Research Methodology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- FreiDok plus (Universitätsbibliothek Freiburg) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
- Keywords
- Google Scholar, systematic search, search recall, search precision, search strategies, MEDLINE
- MeSH terms
- Abstracting and Indexing, Algorithms, Humans, Review Literature as Topic, MEDLINE, Internet, Data Mining, Search Engine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
