The comparative recall of Google Scholar versus PubMed in identical searches for biomedical systematic reviews: a review of searches used in systematic reviews
Open accessAbstract
BACKGROUND: The usefulness of Google Scholar (GS) as a bibliographic database for biomedical systematic review (SR) searching is a subject of current interest and debate in research circles. Recent research has suggested GS might even be used alone in SR searching. This assertion is challenged here by testing whether GS can locate all studies included in 21 previously published SRs. Second, it examines the recall of GS, taking into account the maximum number of items that can be viewed, and tests whether more complete searches created by an information specialist will improve recall compared to the searches used in the 21 published SRs. METHODS: The authors identified 21 biomedical SRs that had used GS and PubMed as information sources and reported their use of identical, reproducible search strategies in both databases. These search strategies were rerun in GS and PubMed, and analyzed as to their coverage and recall. Efforts were made to improve searches that underperformed in each database. RESULTS: GS' overall coverage was higher than PubMed (98% versus 91%) and overall recall is higher in GS: 80% of the references included in the 21 SRs were returned by the original searches in GS versus 68% in PubMed. Only 72% of the included references could be used as they were listed among the first 1,000 hits (the maximum number shown). Practical precision (the number of included references retrieved in the first 1,000, divided by 1,000) was on average 1.9%, which is only slightly lower than in other published SRs. Improving searches with the lowest recall resulted in an increase in recall from 48% to 66% in GS and, in PubMed, from 60% to 85%. CONCLUSIONS: Although its coverage and precision are acceptable, GS, because of its incomplete recall, should not be used as a single source in SR searching. A specialized, curated medical database such as PubMed provides experienced searchers with tools and functionality that help improve recall, and numerous options in order to optimize precision. Searches for SRs should be performed by experienced searchers creating searches that maximize recall for as many databases as deemed necessary by the search expert.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Previous studies have also calculated precision and recall of search systems from data reported by specific evidence-synthesis studies.
“Previous studies have also calculated precision and recall of search systems from data reported by specific evidence‐synthesis studies.24, 32, 33, 34”
Quality criteria have previously been used to evaluate search systems in terms of recall and precision.
“Quality criteria have previously been used to evaluate search systems in terms of recall and precision indicating their suitability of search in general and systematic search in particular.”
The distinction between principal and supplementary resources for systematic reviews was also used in previous assessments of search system qualities.
“The distinction between principal and supplementary resources for systematic reviews was also used in previous assessments of search system qualities.33, 57”
Until now, studies have largely examined the suitability of search systems for certain scholarly tasks for individual systems or by comparing a few systems.
“Until now, studies have largely examined the suitability of search systems for certain scholarly tasks for individual systems or by comparing a few systems.33, 50, 95”
Google Scholar's coverage and recall is an inadequate reason to use it as principal search system in systematic searches.
“Our findings support the criticism of Bramer et al,33 Bramer et al,34 and Boeker et al25 and indicate that Google Scholar's coverage and recall is an inadequate reason to use it as principal search system in systematic searches.53”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & for the PRISMA Group (2009) · BMJ · doi:10.1136/bmj.b2535 · Cited by 83,823 · see citing works
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The PRISMA Group (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000097 · Cited by 64,042 · see citing works
Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & the PRISMA Group* (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-151-4-200908180-00135 · Cited by 37,554 · see citing works
Cochrane Handbook for Systematic Reviews of Interventions Version 5.0.1. The Cochrane Collaboration
Higgins, J., Green, S. R., & Higgins, J. (2008) · Cited by 5,544 · see citing works
Comparison of PubMed, Scopus, Web of Science, and Google Scholar: strengths and weaknesses
Falagas, M. E., Pitsouni, E. I., Malietzis, G. A., & Pappas, G. (2008) · The FASEB Journal · doi:10.1096/fj.07-9492lsf · Cited by 5,039 · see citing works
Is the coverage of google scholar enough to be used alone for systematic reviews
Gehanno, J.-F., Rollin, L., & Darmoni, S. (2013) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-13-7 · Cited by 357 · see citing works
Systematic reviews need systematic searchers.
(2005) · Europe PMC (PubMed Central) · Cited by 257 · see citing works
Comparing test searches in PubMed and Google Scholar
Shultz, M. (2007) · Journal of the Medical Library Association JMLA · doi:10.3163/1536-5050.95.4.442 · Cited by 231 · see citing works
Boeker, M., Vach, W., & Motschall, E. (2013) · BMC Medical Research Methodology · doi:10.1186/1471-2288-13-131 · Cited by 225 · see citing works
Tober, M. (2011) · Medical Laser Application · doi:10.1016/j.mla.2011.05.006 · Cited by 192 · see citing works
Google Scholar is not enough to be used alone for systematic reviews
Giustini, D., & Kamel Boulos, M. N. (2013) · Online Journal of Public Health Informatics · doi:10.5210/ojphi.v5i2.4623 · Cited by 187 · see citing works
Self-citationShould meta-analysts search Embase in addition to Medline?
Sampson, M. (2003) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(03)00110-0 · Cited by 160 · see citing works
Retrieving Clinical Evidence: A Comparison of PubMed and Google Scholar for Quick Clinical Searches
Shariff, S. Z., Bejaimal, S. A. D., Sontrop, J. M., Iansavichus, A. V., Haynes, R. B., Weir, M. A., & Garg, A. X. (2013) · Journal of Medical Internet Research · doi:10.2196/jmir.2624 · Cited by 148 · see citing works
A longitudinal study of Google Scholar coverage between 2012 and 2013
Harzing, A.-W. K. (2013) · Scientometrics · doi:10.1007/s11192-013-0975-y · Cited by 141 · see citing works
Searching one or two databases was insufficient for meta-analysis of observational studies
Lemeshow, A. R., Blum, R. E., Berlin, J. A., Stoto, M. A., & Colditz, G. A. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2005.03.004 · Cited by 137 · see citing works
How Google is changing medicine
Giustini, D. (2005) · BMJ · doi:10.1136/bmj.331.7531.1487 · Cited by 118 · see citing works
Self-citationComparison of PubMed and Google Scholar Literature Searches
Anders, M., & Evans, D. P. (2010) · Respiratory Care · doi:10.4187/respcare.10550578 · Cited by 117 · see citing works
Medical literature searches: a comparison of P ub M ed and G oogle S cholar
Nourbakhsh, E., Nugent, R., Wang, H., Çevik, C., & NUGENT, K. M. (2012) · Health Information & Libraries Journal · doi:10.1111/j.1471-1842.2012.00992.x · Cited by 113 · see citing works
Active Assistance Technology for Health-Related Behavior Change: An Interdisciplinary Review
Kennedy, C. M., powell, J. A., Payne, T. H., Ainsworth, J. D., Boyd, A., & Buchan, I. E. (2012) · Journal of Medical Internet Research · doi:10.2196/jmir.1893 · Cited by 111 · see citing works
Poor reporting and inadequate searches were apparent in systematic reviews of adverse effects
Golder, S., Loke, Y. K., & McIntosh, H. M. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.06.005 · Cited by 97 · see citing works
Which resources should be used to identify RCT/CCTs for systematic reviews: a systematic review
Crumley, E. T., Wiebe, N., Cramer, K., Klassen, T. P., & Hartling, L. (2005) · BMC Medical Research Methodology · doi:10.1186/1471-2288-5-24 · Cited by 95 · see citing works
Google Scholar Search Performance: Comparative Recall and Precision
Walters, W. H. (2009) · portal Libraries and the Academy · doi:10.1353/pla.0.0034 · Cited by 93 · see citing works
Betrán, A. P., Say, L., Gülmezoglu, A. M., Allen, T., & Hampson, L. (2005) · BMC Medical Research Methodology · doi:10.1186/1471-2288-5-6 · Cited by 83 · see citing works
Precision of healthcare systematic review searches in a cross‐sectional sample
Sampson, M., Tetzlaff, J., & Urquhart, C. (2011) · Research Synthesis Methods · doi:10.1002/jrsm.42 · Cited by 80 · see citing works
Google Scholar: A source for clinicians?
Henderson, J. (2005) · Canadian Medical Association Journal · doi:10.1503/cmaj.050404 · Cited by 76 · see citing works
Google Scholar's Dramatic Coverage Improvement Five Years after Debut
Chen, X. (2010) · Serials Review · doi:10.1016/j.serrev.2010.08.002 · Cited by 66 · see citing works
Google Scholar Versus PubMed in Locating Primary Literature to Answer Drug-Related Questions
Freeman, M. K., Lauderdale, S. A., Kendrach, M. G., & Woolley, T. W. (2009) · Annals of Pharmacotherapy · doi:10.1345/aph.1l223 · Cited by 63 · see citing works
Topfer, L.-A., Parada, A., Menon, D., Noorani, H. Z., Perras, C., & Serra‐Prat, M. (1999) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462399015226 · Cited by 59 · see citing works
Watson, R., & Richardson, P. H. (1999) · British Journal of Medical Psychology · doi:10.1348/000711299160220 · Cited by 53 · see citing works
Sackett, D. L. (1994) · ACP Journal Club · doi:10.7326/acpjc-1994-120-3-a11 · Cited by 49 · see citing works
Haase, A., Follmann, M., Skipka, G., & Kirchner, H. (2007) · BMC Medical Research Methodology · doi:10.1186/1471-2288-7-28 · Cited by 36 · see citing works
Mastrangelo, G., Fadda, E., Rossi, C. R., Zamprogno, E., Buja, A., & Cegolon, L. (2010) · BMC Research Notes · doi:10.1186/1756-0500-3-131 · Cited by 21 · see citing works
Flagged first, then by citations
1 of the 33 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Gusenbauer, M., & Haddaway, N. R. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1378 · Cited by 2,379 · see citing works
“33,50,95 Our methods allowed a comprehensive review of many different search systems.”
BackgroundMethodResultBramer, W. M., Rethlefsen, M. L., Kleijnen, J., & Franco, O. H. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0644-y · Cited by 2,114 · see citing works
“Nevertheless others have concluded that a single database is not sufficient to retrieve all references for systematic reviews [18, 19].”
BackgroundSelf-citationA systematic approach to searching: an efficient and complete method to develop literature searches
Bramer, W. M., De Jonge, G. B., Rethlefsen, M. L., Mast, F., & Kleijnen, J. (2018) · Journal of the Medical Library Association · doi:10.5195/jmla.2018.283 · Cited by 863 · see citing works
Self-citationHalevi, G., Moed, H., & Bar-Ilan, J. (2017) · Journal of Informetrics · doi:10.1016/j.joi.2017.06.005 · Cited by 488 · see citing works
Yao, X. I., Wang, X., Speicher, P. J., Hwang, E. S., Cheng, P., Harpole, D. H., Berry, M. F., Schrag, D., & Pang, H. (2017) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/djw323 · Cited by 388 · see citing works
Searching for studies: a guide to information retrieval for Campbell systematic reviews
Kugley, S., Wade, A., Thomas, J., Mahood, Q., Jørgensen, A. K., Hammerstrøm, K., & Sathe, N. (2017) · Campbell Systematic Reviews · doi:10.4073/cmg.2016.1 · Cited by 371 · see citing works
Rethlefsen, M. L., Farrell, A. M., Osterhaus Trzasko, L. C., & Brigham, T. J. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.11.025 · Cited by 331 · see citing works
Eight problems with literature reviews and how to fix them
Haddaway, N., Bethel, A. C., Dicks, L. V., Koricheva, J., Macura, B., Petrokofsky, G., Pullin, A. S., Savilaakso, S., & Stewart, G. B. (2020) · Nature Ecology & Evolution · doi:10.1038/s41559-020-01295-x · Cited by 280 · see citing works
A Review of Food–Drug Interactions on Oral Drug Absorption
Deng, J., Zhu, X., Chen, Z., Fan, C. H., Kwan, H. S., Wong, C. H., Shek, K. Y., Zuo, Z., & Lam, T. N. (2017) · Drugs · doi:10.1007/s40265-017-0832-z · Cited by 227 · see citing works
Bramer, W. M., Giustini, D., & Kramer, B. M. R. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0215-7 · Cited by 202 · see citing works
Self-citationCan we use Google Scholar to identify highly-cited documents?
Martín‐Martín, A., Orduña‐Malea, E., Harzing, A.-W. K., & López‐Cózar, E. D. (2016) · Journal of Informetrics · doi:10.1016/j.joi.2016.11.008 · Cited by 183 · see citing works
Systematic searching for environmental evidence using multiple tools and sources
Livoreil, B., Glanville, J., Haddaway, N. R., Bayliss, H., Bethel, A., de Lachapelle, F. F., Robalino, S., Savilaakso, S., Zhou, W., Petrokofsky, G., et al. (2017) · Environmental Evidence · doi:10.1186/s13750-017-0099-6 · Cited by 156 · see citing works
Roles for librarians in systematic reviews: a scoping review
Spencer, A. J., & Eldredge, J. D. (2018) · Journal of the Medical Library Association JMLA · doi:10.5195/jmla.2018.82 · Cited by 157 · see citing works
Conduct and reporting of citation searching in Cochrane systematic reviews: A cross‐sectional study
Briscoe, S., Bethel, A. C., & Rogers, M. (2019) · Research Synthesis Methods · doi:10.1002/jrsm.1355 · Cited by 151 · see citing works
Pereza, N., Ostojić, S., Kapović, M., & Peterlin, B. (2016) · Fertility and Sterility · doi:10.1016/j.fertnstert.2016.10.007 · Cited by 149 · see citing works
Cooper, C., Booth, A., Britten, N., & Garside, R. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0625-1 · Cited by 137 · see citing works
Stansfield, C., Dickson, K., & Bangpan, M. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0371-9 · Cited by 122 · see citing works
Lutfiyya, M. N., Chang, L., McGrath, C. A., Dana, C., & Lipsky, M. S. (2019) · PLoS ONE · doi:10.1371/journal.pone.0218578 · Cited by 120 · see citing works
The contribution of databases to the results of systematic reviews: a cross-sectional study
Hartling, L., Featherstone, R., Nuspl, M., Shave, K., Dryden, D. M., & Vandermeer, B. (2016) · BMC Medical Research Methodology · doi:10.1186/s12874-016-0232-1 · Cited by 118 · see citing works
Looman, W. M., Huijsman, R., & Fabbricotti, I. N. (2018) · Health & Social Care in the Community · doi:10.1111/hsc.12571 · Cited by 116 · see citing works
Searching for studies: A guide to information retrieval for Campbell systematic reviews
MacDonald, H. R., Comer, C., Foster, M., Labelle, P., Marsalis, S. E., Nyhan, K., Premji, Z., Rogers, M., Splenda, R., Stansfield, C., et al. (2024) · Campbell Systematic Reviews · doi:10.1002/cl2.1433 · Cited by 105 · see citing works
A scoping review of opioid misuse in the rural United States
Palombi, L. C., Hill, C. A. S., Lipsky, M. S., Swanoski, M. T., & Lutfiyya, M. N. (2018) · Annals of Epidemiology · doi:10.1016/j.annepidem.2018.05.008 · Cited by 91 · see citing works
Gusenbauer, M., & Haddaway, N. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1457 · Cited by 88 · see citing works
The Structure and Conduct of a Narrative Literature Review
Pautasso, M. (2019) · A Guide to the Scientific Career · doi:10.1002/9781118907283.ch31 · Cited by 124 · see citing works
Multidisciplinary Simulation-Based Team Training for Trauma Resuscitation: A Scoping Review
McLaughlin, C. M., Barry, W. E., Barin, E. N., Kysh, L., Auerbach, M. A., Upperman, J. S., Burd, R. S., & Jensen, A. R. (2019) · Journal of surgical education · doi:10.1016/j.jsurg.2019.05.002 · Cited by 73 · see citing works
Aliabadi, A., Sheikhtaheri, A., & Ansari, H. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa186 · Cited by 68 · see citing works
Linder, S. K., Kamath, G. R., Pratt, G. F., Saraykar, S. S., & Volk, R. J. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.10.008 · Cited by 63 · see citing works
Teledentistry Implementation During the COVID-19 Pandemic: Scoping Review
Hung, M., Lipsky, M. S., Phuatrakoon, T. N., Nguyen, M., Licari, F. W., & Unni, E. J. (2022) · Interactive Journal of Medical Research · doi:10.2196/39955 · Cited by 61 · see citing works
Database selection in systematic reviews: an insight through clinical neurology
Vassar, M. B., Yerokhin, V., Sinnett, P., Weiher, M., Muckelrath, H. D., Carr, B., Varney, L., & Cook, G. (2017) · Health Information & Libraries Journal · doi:10.1111/hir.12176 · Cited by 61 · see citing works
Evans, S. C., Amaro, C. M., Herbert, R. S., Blossom, J. B., & Roberts, M. C. (2018) · PLoS ONE · doi:10.1371/journal.pone.0192219 · Cited by 57 · see citing works
Structural aspects of corporate governance and family firm performance: a systematic review
Azila-Gbettor, E. M., Honyenuga, B. Q., Berent‐Braun, M. M., & Kil, A. J. (2018) · Journal of Family Business Management · doi:10.1108/jfbm-12-2017-0045 · Cited by 54 · see citing works
Levay, P., Ainsworth, N., Kettle, R., & Morgan, A. (2016) · Research Synthesis Methods · doi:10.1002/jrsm.1158 · Cited by 50 · see citing works
Competing stimulus assessments: A systematic review
Haddock, J. N., & Hagopian, L. P. (2020) · Journal of Applied Behavior Analysis · doi:10.1002/jaba.754 · Cited by 48 · see citing works
Zubir, M. Z., Noor, A. A., Rizal, A. M., Harith, A. A., Abas, M. I., Zakaria, Z., & Bakar, A. F. A. (2024) · PLoS ONE · doi:10.1371/journal.pone.0293694 · Cited by 47 · see citing works
Frandsen, T. F., Eriksen, M. B., Hammer, D. M. G., & Christensen, J. B. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.04.015 · Cited by 44 · see citing works
Carbajal-de-la-Fuente, A. L., Sánchez-Casaccia, P., Piccinali, R. V., Provecho, Y. M., Salvá, L., Meli, S., Cano, F., Hernández, R., & Nattero, J. (2022) · PLoS neglected tropical diseases · doi:10.1371/journal.pntd.0011003 · Cited by 44 · see citing works
Bates, J. A., Best, P., McQuilkin, J., & Taylor, B. J. (2016) · The Journal of Academic Librarianship · doi:10.1016/j.acalib.2016.11.003 · Cited by 44 · see citing works
Satellite Remote Sensing in Shark and Ray Ecology, Conservation and Management
Williamson, M. J., Tebbs, E. J., Dawson, T. P., & Jacoby, D. M. P. (2019) · Frontiers in Marine Science · doi:10.3389/fmars.2019.00135 · Cited by 39 · see citing works
Sülz, S., van Elten, H. J., Askari, M., Weggelaar‐Jansen, A. M., & Huijsman, R. (2021) · Journal of Medical Internet Research · doi:10.2196/24363 · Cited by 40 · see citing works
Development of an efficient search filter to retrieve systematic reviews from PubMed
Oliván, J. A. S., Cuenca, G. M., & Avilés, R. A. (2021) · Journal of the Medical Library Association JMLA · doi:10.5195/jmla.2021.1223 · Cited by 38 · see citing works
Associations Between Cigarette Smoking and Pain Among Veterans
Chapman, S. L. C., & Wu, L.-T. T. (2015) · Epidemiologic Reviews · doi:10.1093/epirev/mxu008 · Cited by 37 · see citing works
Variation in number of hits for complex searches in Google Scholar
Bramer, W. M. (2016) · Journal of the Medical Library Association JMLA · doi:10.3163/1536-5050.104.2.009 · Cited by 35 · see citing works
Self-citationA systematic literature review on the enablers of green marketing adoption: Consumer perspective
Kumari, R., Verma, R., Debata, B. R., & Ting, H. (2022) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2022.132852 · Cited by 41 · see citing works
Flint, A. E., Waterman, M. G., Bowmer, G., Vadlamani, G., Chumas, P., & Morrall, M. C. H. J. (2017) · Seizure · doi:10.1016/j.seizure.2017.09.011 · Cited by 34 · see citing works
Reviewing the influence of positive leadership on worker well-being: A comprehensive analysis
Azila-Gbettor, E. M., Honyenuga, B. Q., Atatsi, E. A., Laryea, C. N. A., & Quarshie, A. N. K. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e24134 · Cited by 54 · see citing works
Happell, B., Ewart, S. B., Platania‐Phung, C., & Stanton, R. (2016) · International Journal of Mental Health Nursing · doi:10.1111/inm.12226 · Cited by 30 · see citing works
A Comparative Study of Perceptions and Use of Google Scholar and Academic Library Discovery Systems
Oh, K., & Colón‐Aguirre, M. (2019) · College & Research Libraries · doi:10.5860/crl.80.6.876 · Cited by 34 · see citing works
Martin, M. M., Baker, N. C., Boyes, W. K., Carstens, K. E., Culbreth, M. E., Gilbert, M. E., Harrill, J. A., Nyffeler, J., Padilla, S., Friedman, K. P., et al. (2022) · Neurotoxicology and Teratology · doi:10.1016/j.ntt.2022.107117 · Cited by 29 · see citing works
Complementary approaches to searching MEDLINE may be sufficient for updating systematic reviews
Sampson, M., de Bruijn, B., Urquhart, C. J., & Shojania, K. G. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.03.004 · Cited by 28 · see citing works
Volumetric Muscle Loss: A Bibliometric Analysis of a Decade of Progress
Kulwatno, J., Goldman, S. M., & Dearth, C. L. (2022) · Tissue Engineering Part B Reviews · doi:10.1089/ten.teb.2022.0150 · Cited by 27 · see citing works
Morshed, T., & Hayden, S. R. (2019) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2019.07.003 · Cited by 24 · see citing works
Vermeulen-Oskam, E., Franklin, C., Hof, L. P. M. V., Stams, G. J. J. M., van Vugt, E., Assink, M., Veltman, E., Froerer, A. S., Staaks, J. P. C., & Zhang, A. (2024) · Clinical Psychology Review · doi:10.1016/j.cpr.2024.102512 · Cited by 30 · see citing works
Deri, S., Rappaz, J., Aiello, L. M., & Quercia, D. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274312 · Cited by 23 · see citing works
Erlangga, E., Machuku, O., & Dahino, C. J. (2023) · Cogent Food & Agriculture · doi:10.1080/23311932.2023.2273634 · Cited by 22 · see citing works
Helicopter Emergency Medical Services Literature 1972-2017: Characteristics and Trends
Thomas, S. H., Thomas, S. H., Thomas, S. A., & Pathan, S. A. (2019) · Air Medical Journal · doi:10.1016/j.amj.2018.11.016 · Cited by 21 · see citing works
Langenberger, B., Thoma, A., & Vogt, V. (2022) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-022-01751-7 · Cited by 21 · see citing works
Sodi, T., Quarshie, E. N., Asante, K. O., Radzilani-Makatu, M., Makgahlela, M., Nkoana, S. E., & Mutambara, J. (2022) · BMJ Open · doi:10.1136/bmjopen-2022-063687 · Cited by 24 · see citing works
Comparative analysis of Cochrane and non-Cochrane reviews over three decades
Andersen, M. Z., Zeinert, P., Rosenberg, J., & Fonnes, S. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02531-2 · Cited by 19 · see citing works
Information sources for obesity prevention policy research: a review of systematic reviews
Hanneke, R., & Young, S. K. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0543-2 · Cited by 18 · see citing works
Pulikowski, A., & Matysek, A. (2021) · The Journal of Academic Librarianship · doi:10.1016/j.acalib.2021.102417 · Cited by 19 · see citing works
Reynders, R. M., Ladu, L., Ronchi, L., Girolamo, N. D., de Lange, J., Roberts, N. W., & Plüddemann, A. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0014-6 · Cited by 17 · see citing works
The burden they bear: A scoping review of physician empathy in the intensive care unit
Bunin, J. L., Shohfi, E. E., Meyer, H. S., Ely, E. W., & Varpio, L. (2021) · Journal of Critical Care · doi:10.1016/j.jcrc.2021.05.014 · Cited by 16 · see citing works
The ethics of scientific recommender systems
Polonioli, A. (2020) · Scientometrics · doi:10.1007/s11192-020-03766-1 · Cited by 16 · see citing works
Anderson, E. J., McClelland, J. D., Krause, C. M., Krause, K. C., Garcia, D. O., & Koss, M. P. (2019) · BMJ Open · doi:10.1136/bmjopen-2019-029880 · Cited by 13 · see citing works
MEDLINE search retrieval issues: A longitudinal query analysis of five vendor platforms
Burns, C. S., Nix, T., Shapiro, R. M., & Huber, J. T. (2021) · PLoS ONE · doi:10.1371/journal.pone.0234221 · Cited by 13 · see citing works
Methodische Ansätze zu Evidenzsynthesen in der Bildungstechnologie: Eine tertiäre Übersichtsarbeit
Buntins, K., Bedenlier, S., Marín, V., Händel, M., & Bond, M. (2023) · MedienPädagogik: Zeitschrift für Theorie und Praxis der Medienbildung · doi:10.21240/mpaed/54/2023.12.20.x · Cited by 13 · see citing works
An analysis of the fate of 917 manuscripts rejected from Clinical Otolaryngology
Earnshaw, C. H., Edwin, C., Bhat, J., Krishnan, M., Mamais, C., Somashekar, S., Sunil, A., Williams, S. P., & Leong, S. C. (2016) · Clinical Otolaryngology · doi:10.1111/coa.12820 · Cited by 13 · see citing works
On the pitfalls of bibliographic database searching: comparing successful and less successful users
Rosman, T., Mayer, A., & Krampen, G. (2015) · Behaviour and Information Technology · doi:10.1080/0144929x.2015.1066446 · Cited by 12 · see citing works
Brandsen, A., & Lippok, F. E. (2021) · Journal of Archaeological Science · doi:10.1016/j.jas.2021.105456 · Cited by 12 · see citing works
Reynders, R. M., Ronchi, L., Ladu, L., Girolamo, N. D., de Lange, J., Roberts, N. W., & Mickan, S. M. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0198-4 · Cited by 12 · see citing works
Coloring in the Links: Capturing Social Ties as They are PerceivedPreprint
Deri, S., Rappaz, J., Aiello, L. M., & Quercia, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.04528 · Cited by 11 · see citing works
Eybye, M. N., Madsen, S. D., Schultz, A. N. Ø., & Nim, C. G. (2022) · Chiropractic & Manual Therapies · doi:10.1186/s12998-022-00468-8 · Cited by 10 · see citing works
Azila-Gbettor, E. M., Honyenuga, B. Q., Blomme, R. J., & Kil, A. J. (2021) · Journal of Family Business Management · doi:10.1108/jfbm-03-2020-0023 · Cited by 10 · see citing works
Search results outliers among MEDLINE platforms
Burns, C. S., Shapiro, R. M., Nix, T., & Huber, J. T. (2019) · Journal of the Medical Library Association JMLA · doi:10.5195/jmla.2019.622 · Cited by 9 · see citing works
Literature review in biomedical research: useful search engines beyond PubMed
Cals, J., & Kotz, D. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.10.012 · Cited by 9 · see citing works
Azila-Gbettor, E. M., Mensah, C., Quarshie, A. N. K., & Abiemo, M. K. (2024) · Cogent Education · doi:10.1080/2331186x.2024.2331270 · Cited by 12 · see citing works
Google Scholar as a Resource for Systematic Reviews in Clinical Medicine
Falagas, M. E., Paliogianni, P. M., Kontogiannis, D. S., Ragias, D., & Johnson, E. O. (2025) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.70206 · Cited by 8 · see citing works
Information Retrieval in Telemedicine: a Comparative Study on Bibliographic Databases
Ahmadi, M., Sarabi, R. E., Orak, R. J., & Bahaadinbeigy, K. (2015) · Acta Informatica Medica · doi:10.5455/aim.2015.23.172-176 · Cited by 8 · see citing works
A plea for minimally biased naturalistic philosophy
Polonioli, A. (2017) · Synthese · doi:10.1007/s11229-017-1628-0 · Cited by 10 · see citing works
Apriyani, V., Holle, M. J. M., Silangen, C., Oktalira, F. T., & Mumbunan, S. (2021) · Environmental Evidence · doi:10.1186/s13750-021-00233-z · Cited by 7 · see citing works
The drivers of anguillid eel movement in lentic water bodies: a systematic map
Williamson, M. J., Jacoby, D. M. P., & Piper, A. T. (2023) · Reviews in Fish Biology and Fisheries · doi:10.1007/s11160-022-09751-6 · Cited by 7 · see citing works
A Systematic Review on the Sufficiency of PubMed and Google Scholar for Biosciences
Teo, Y. H., & Ling, M. H. (2020) · Acta Scientific Medical Sciences · doi:10.31080/asms.2020.04.0786 · Cited by 6 · see citing works
Comment utiliserPubMedpour les recherches bibliographiques en éducation thérapeutique du patient
de Andrade, V. (2014) · Education thérapeutique du patient - Therapeutic patient education · doi:10.1051/tpe/2014011 · Cited by 7 · see citing works
PubMed Central archiving: a major milestone for a scholarly journal
Gasparyan, A. Y., & Kitas, G. D. (2019) · Mediterranean Journal of Rheumatology · doi:10.31138/mjr.31.1.3 · Cited by 6 · see citing works
Glove breach occurrence during surgical procedures: the benefits of double/indicator system gloves
Rippon, M. G., Rogers, A. A., & Ousey, K. J. (2025) · Journal of Hospital Infection · doi:10.1016/j.jhin.2025.04.010 · Cited by 5 · see citing works
Dittus, M., Aiello, L. M., & Quercia, D. (2017) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3134674 · Cited by 5 · see citing works
Resende, L. C., do Nascimento, M. M. G., Barbosa, M. M., Rezende, C. D. P., Pantuzza, L. L. N., & Reis, E. A. (2022) · Pharmacy · doi:10.3390/pharmacy10060151 · Cited by 5 · see citing works
Learning Experimental Design through Targeted Student-Centric Journal Club with Screencasting.
Carter, B. S., Hamilton, D. E., & Thompson, R. C. (2017) · PubMed Central · Cited by 5 · see citing works
Smith, A. F., Klotz, A., & Wormstone, I. M. (2016) · Drug Discovery Today · doi:10.1016/j.drudis.2016.01.001 · Cited by 4 · see citing works
Thiamine (vitamin B1) as an insect repellent: a scoping review
Shelomi, M. (2022) · Bulletin of Entomological Research · doi:10.1017/s0007485321001176 · Cited by 4 · see citing works
Tawfique, M., Alam, A., Ergler, C. R., & Freeman, C. (2025) · City Culture and Society · doi:10.1016/j.ccs.2025.100641 · Cited by 3 · see citing works
Kognitive Prozesse in Screening-Prozessen – Suchstrategien in der Bildungstechnologieforschung
Buntins, K., Mulders, M., & Schröder, N. (2023) · MedienPädagogik Zeitschrift für Theorie und Praxis der Medienbildung · doi:10.21240/mpaed/54/2023.11.19.x · Cited by 3 · see citing works
Borycz, J., Carroll, A. J., & Eskridge, H. N. (2021) · Journal of eScience Librarianship · doi:10.7191/jeslib.2021.1192 · Cited by 3 · see citing works
Antiphospholipid syndrome, thrombosis, and vaccination in the COVID-19 pandemic
Zekić, T., & Belančić, A. (2024) · Rheumatology International · doi:10.1007/s00296-023-05531-y · Cited by 4 · see citing works
Database choices in endocrine systematic reviews
Vassar, M. B., Carr, B., Kash-Holley, M., DeWitt, E., Koller, C., Day, J. M., Day, K., Herrmann, D., & Holzmann, M. (2015) · Journal of the Medical Library Association JMLA · doi:10.3163/1536-5050.103.4.005 · Cited by 3 · see citing works
Covington, S. N. (2015) · Cambridge University Press eBooks · doi:10.1017/cbo9781107449398.020 · Cited by 3 · see citing works
Rebuttal to Winn et al. (2024) ’s “Redefining searching in non-medical sciences systematic reviews”
Lipke, L., Monnin, J., & Fonseca, C. (2025) · Journal of Librarianship and Information Science · doi:10.1177/09610006251387675 · Cited by 2 · see citing works
Search tools and scholarly citation practices in literary studies
Lehnen, C. A., & Insua, G. (2022) · Reference Services Review · doi:10.1108/rsr-07-2022-0025 · Cited by 3 · see citing works
The Application of AI in Oncology Research in Türkiye: Impact and Future Directions
Osman, R., & Arslan, H. (2025) · Gazi University Journal of Science Part A Engineering and Innovation · doi:10.54287/gujsa.1768020 · Cited by 2 · see citing works
A Replication of Failure, Not a Failure to Replicate
Holden, G. W., Barker, K., Kuppens, S., Rosenberg, G., & LeBreton, J. (2014) · Research on Social Work Practice · doi:10.1177/1049731514530000 · Cited by 2 · see citing works
- Authors
- Bramer, W. M., Giustini, D., Kramer, B. M., & Anderson, P.
- Affiliations
- Erasmus MC (NL)
- University of British Columbia (CA)
- Utrecht University (NL)
- Corresponding author
- Bramer, Wichor M
- Venue
- Systematic Reviews
- Published
- 2013-12
- Volume
- 2
- Issue
- 1
- Pages
- 115-115
- ISSN
- 2046-4053 (ISSN-L)
- PMC
- PMC3882110
- OpenAlex
- W2100684744
- Cited by
- 139
- References
- 33
- Citations by year
- 2026: 5 · 2025: 11 · 2024: 11 · 2023: 5 · 2022: 16 · 2021: 13 · 2020: 10 · 2019: 17 · 2018: 7 · 2017: 15 · 2016: 15 · 2015: 10 · 2014: 4
- FWCI
- 1.98
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Systematic Reviews (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePub (Erasmus University Rotterdam) (submittedVersion, other-oa)
- cIRcle (University of British Columbia) (submittedVersion, cc-by)
- Deep Blue (University of Michigan) (submittedVersion)
- RePub (Erasmus University Rotterdam) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
- Keywords
- Google Scholar, PubMed, search precision, information retrieval, database comparison
- MeSH terms
- Systematic Reviews as Topic, Humans, Information Storage and Retrieval, Internet, PubMed, Search Engine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
