A checklist to assess database‐hosting platforms for designing and running searches for systematic reviews
Open accessAbstract
Background: Systematic reviews require literature searches that are precise, sensitive and often complex. Database‐hosting platforms need to facilitate this type of searching in order to minimise errors and the risk of bias in the results. Objectives: The main objective of the study was to create a generic checklist of criteria to assess the ability of host platforms to cope with complex searching, for example, for systematic reviews, and to test the checklist against three host platforms (EBSCO host, Ovid SP and ProQuest). Method: The checklist was developed as usual review work was carried out and through discussion between the two authors. Attributes on the checklist were designated as ‘desirable’ or ‘essential’. The authors tested the checklist independently against three host platforms and graded their performance from 1 (insufficient) to 3 (performs well). Results: Fifty‐five desirable or essential attributes were identified for the checklist. None of the platforms performed well for all of the attributes on the checklist. Conclusions: Not all database‐hosting platforms are designed for complex searching. Librarians and other decision‐makers who work in health research settings need to be aware of the different limitations of host platforms for complex searching when they are making purchasing decisions or training others.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Previous studies examining the suitability of search systems for evidence synthesis have focused on a limited number of search systems and/or based their analysis on a review of the search interface, yet without any in-depth examination of core functionalities that allow reliable query-based searching.
“Previous studies examining the suitability of search systems for evidence synthesis have focused on a limited number of search systems and/or based their analysis on a review of the search interface, yet without any in‐depth examination of core functionalities that allow reliable query‐based searching.19, 30, 31, 32”
Previous studies have also calculated precision and recall of search systems from data reported by specific evidence-synthesis studies.
“Previous studies have also calculated precision and recall of search systems from data reported by specific evidence‐synthesis studies.24, 32, 33, 34”
Cites
Systematic reviews: CRD's guidance for undertaking reviews in health care
Tacconelli, E. (2010) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(10)70065-7 · Cited by 3,237 · see citing works
Papaioannou, D., Sutton, A., Carroll, C., Booth, A., & Wong, R. (2010) · Health Information & Libraries Journal · doi:10.1111/j.1471-1842.2009.00863.x · Cited by 318 · see citing works
Savoie, I., Helmer, D., Green, C. J., & Kazanjian, A. (2003) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462303000163 · Cited by 105 · see citing works
Searching for information on outcomes: do you need to be comprehensive?
Brettle, A., Long, A., Grant, M. J., & Greenhalgh, J. (1998) · BMJ Quality & Safety · doi:10.1136/qshc.7.3.163 · Cited by 47 · see citing works
Younger, P., & Boddy, K. (2009) · Health Information & Libraries Journal · doi:10.1111/j.1471-1842.2008.00785.x · Cited by 43 · see citing works
Comparing Four CAB Abstracts Platforms from a Veterinary Medicine Perspective
Sewell, R. R. (2011) · Journal of Electronic Resources in Medical Libraries · doi:10.1080/15424065.2011.576608 · Cited by 8 · see citing works
Examining Biological Abstracts on Two Platforms: What Do End Users Need to Know?
Bandyopadhyay, A. (2010) · Science & Technology Libraries · doi:10.1080/01942620903579377 · Cited by 7 · see citing works
The GeoRef Database: A Detailed Comparison and Analysis of Three Platforms
Kimball, R. (2010) · Science & Technology Libraries · doi:10.1080/01942620903579419 · Cited by 5 · see citing works
Casler, C. L., Herring, E., Smith, H., Moberly, H. K., Flood, S., & Perry, V. (2002) · Journal of Agricultural & Food Information · doi:10.1300/j108v04n03_05 · Cited by 1 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Gusenbauer, M., & Haddaway, N. R. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1378 · Cited by 2,379 · see citing works
“32-34 This analysis focuses instead on evaluation criteria for systematic reviews across disciplines, following universally accepted conduct guidance (ie, Cochrane, Campbell, and CEE).”
BackgroundRoles for librarians in systematic reviews: a scoping review
Spencer, A. J., & Eldredge, J. D. (2018) · Journal of the Medical Library Association JMLA · doi:10.5195/jmla.2018.82 · Cited by 157 · see citing works
Smith, L. C. (2023) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24851 · Cited by 19 · see citing works
MEDLINE search retrieval issues: A longitudinal query analysis of five vendor platforms
Burns, C. S., Nix, T., Shapiro, R. M., & Huber, J. T. (2021) · PLoS ONE · doi:10.1371/journal.pone.0234221 · Cited by 13 · see citing works
Search strategies (filters) to identify systematic reviews in MEDLINE and Embase
Liquitay, C. M. E., Garegnani, L., Garrote, V., Solà, I., & Franco, J. V. A. (2023) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000054.pub2 · Cited by 12 · see citing works
Handsearching had best recall but poor efficiency when exporting to a bibliographic tool: case study
Cooper, C., Snowsill, T., Worsley, C., Prowse, A., O’Mara-Eves, A., Greenwood, H., Boulton, E., & Strickson, A. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.03.013 · Cited by 11 · see citing works
A Comparison of Selected Bibliographic Database Search Retrieval for Agricultural Information
Ritchie, S., Banyas, K., & Sevin, C. (2019) · Issues in Science and Technology Librarianship · doi:10.29173/istl48 · Cited by 10 · see citing works
Search results outliers among MEDLINE platforms
Burns, C. S., Shapiro, R. M., Nix, T., & Huber, J. T. (2019) · Journal of the Medical Library Association JMLA · doi:10.5195/jmla.2019.622 · Cited by 9 · see citing works
Cooper, C., Court, R. A., Kotas, E., & Schauberger, U. (2021) · Research Synthesis Methods · doi:10.1002/jrsm.1477 · Cited by 10 · see citing works
An analysis of data sources and study registries used in systematic reviews
Nick, J. M., & Sarpy, N. L. (2022) · Worldviews on Evidence-Based Nursing · doi:10.1111/wvn.12614 · Cited by 6 · see citing works
How do search systems impact systematic searching? A qualitative study
Hickner, A. (2023) · Journal of the Medical Library Association JMLA · doi:10.5195/jmla.2023.1647 · Cited by 5 · see citing works
Examining MEDLINE search query reproducibility and resulting variation in search results
Burns, C. S., Shapiro, R. M., Nix, T., & Huber, J. T. (2019) · iConference · doi:10.21900/iconf.2019.103369 · Cited by 3 · see citing works
Methods for Evaluating Database Coverage
Brody, S., Brill, R., & Butcher, R. (2023) · Hypothesis Research Journal for Health Information Professionals · doi:10.18060/27364 · Cited by 2 · see citing works
Rogers, M., Sutton, A., Campbell, F., Whear, R., Bethel, A. C., & Coon, J. T. (2024) · Campbell Systematic Reviews · doi:10.1002/cl2.1380 · Cited by 2 · see citing works
Self-citationKlerings, I., Dobrescu, A. I., Nußbaumer-Streit, B., Sharifan, A., Neubauer-Bruckner, C., & Gartlehner, G. (2026) · BMC Medical Research Methodology · doi:10.1186/s12874-026-02870-9 · Cited by 1 · see citing works
Thirty years of practitioner‐based projects
Grant, M. J. (2014) · Health Information & Libraries Journal · doi:10.1111/hir.12058 · Cited by 1 · see citing works
LIBRARIAN'S ROLE IN DENTISTRY SYSTEMATIC REVIEWS: AN EPIDEMIOLOGIC STUDY
Lehmkuhl, K. M., Silveira, C. Z., Pauletto, P., Savi, M. G. M., Porporatti, A. L., & Canto, G. D. L. (2023) · Perspectivas em Ciência da Informação · doi:10.1590/1981-5344/25776 · Cited by 1 · see citing works
Cooper, C., Brown, A., Court, R. A., & Schauberger, U. (2022) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462322000137 · Cited by 1 · see citing works
Hosting Inspec on Engineering Village or Web of Science
McAllister, J. T., & Diaz, N. S. (2020) · Reference Services Review · doi:10.1108/rsr-08-2019-0050
Methodological Issues with Search in MEDLINE: A Longitudinal Query AnalysisPreprint
Burns, C. S., Nix, T., Shapiro, R. M., & Huber, J. T. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.05.22.110403
Levay, P. (2022) · medRxiv · doi:10.1101/2022.04.23.22274201
Premji, Z., McGill, S., & Riegelman, A. (2026) · Research Synthesis Methods · doi:10.1017/rsm.2026.10101
- Authors
- Bethel, A., & Rogers, M.
- Affiliations
- University of Exeter (GB)
- Corresponding author
- Rogers, Morwenna
- Venue
- Health Information & Libraries Journal
- Publisher
- Wiley
- Published
- 2014-03
- Volume
- 31
- Issue
- 1
- Pages
- 43-53
- ISSN
- 1471-1834 (ISSN-L)
- 1471-1842
- OpenAlex
- W2064716231
- Cited by
- 22
- References
- 9
- Citations by year
- 2026: 2 · 2024: 1 · 2023: 5 · 2022: 3 · 2021: 2 · 2020: 3 · 2019: 4 · 2018: 1 · 2014: 1
- FWCI
- 0.09
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Health Information & Libraries Journal (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- National Institute for Health Research (NIHR)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › Library Science and Information Literacy
- Physical Sciences › Computer Science › Information Systems › Library Science and Information
- Keywords
- systematic reviews, search design, search sensitivity, search precision, EBSCOhost, ProQuest
- MeSH terms
- Humans, Information Systems, Quality Assurance, Health Care, Review Literature as Topic, Databases, Bibliographic, Checklist, Search Engine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
