How good are clinical MEDLINE searches? A comparative study of clinical end-user and librarian searches
The DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
Abstract
The objective of this study was to determine the quality of MEDLINE searches done by physicians, physician trainees, and expert searchers (clinicians and librarians). Its design was an analytic survey with independent replication in a setting of self-service online searching from medical wards, an intensive care unit, a coronary care unit, an emergency room, and an ambulatory clinic in a 300-bed teaching hospital. Participating were all M.D. clinical clerks, house, and attending staff responsible for patients in the above settings. Intervention for all participants consisted of a 2-h small group class and 1-h practice session on MEDLINE searching (GRATEFUL MED) before free access to MEDLINE. Search questions from 104 randomly selected novice searches were given to 1 of 13 clinicians with prior search experience and 1 of 3 librarians to run independent searches (triplicated searches). Measurements and main results from these unique citations of the triplicated searches were sent to expert clinicians to rate for relevance (7-point scale). Recall (number of relevant citations retrieved from an individual search divided by the total number of relevant citations from all searches on the same topic) and precision (proportion of relevant citations retrieved in each search) were calculated. Librarians were significantly better than novices for both. Librarians had equivalent recall to, and better precision than, experienced end-users. Unexpectedly, only 20% of relevant citations were retrieved by more than one search of the set of three, with the conclusion that novice searchers on MEDLINE via GRATEFUL MED after brief training have relatively low recall and precision. Recall improves with experience but precision remains suboptimal. Further research is needed to determine the "learning curve," evaluate training interventions, and explore the non-overlapping retrieval of relevant citations by different searchers.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The skills and experience of the searcher are one of the most important aspects in the effectiveness of systematic review search strategies [23–25].”
From Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study (2017) · cites this paper as [25]
Citations
Cites
Online Access to MEDLINE in Clinical Settings: A Study of Use and Usefulness
Haynes, R. B., McKibbon, K. A., Walker, C. J., Ryan, N., Fitzgerald, D., & Ramsden, M. F. (1990) · Annals of Internal Medicine · doi:10.7326/0003-4819-112-1-78 · Cited by 287
Self-citationWilliamson, J. W., German, P. S., Weiss, R., Skinner, E. A., & Bowes, F. (1989) · Annals of Internal Medicine · doi:10.7326/0003-4819-110-2-151 · Cited by 266
Clinical medical librarianship: a review of the literature.
Cimpl, K. (1985) · PubMed · Cited by 86
Online searching styles: A case‐study‐based model of searching behavior
Fidel, R. (1984) · Journal of the American Society for Information Science · doi:10.1002/asi.4630350404 · Cited by 65
Observations of end-user online searching behavior over eleven years
Sewell, W., & Teitelbaum, S. (1986) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(198607)37:4<234::aid-asi9>3.0.co;2-a · Cited by 65
MEDLARS: report on the evaluation of its operating efficiency
Lancaster, F. W. (1969) · American Documentation · doi:10.1002/asi.4630200204 · Cited by 54
MEDLINE Searching on Colleague:
Kirby, M., & Miller, N. (1986) · Medical Reference Services Quarterly · doi:10.1300/j115v05n03_02 · Cited by 21
Fischhoff, B., & MacGregor, D. G. (1986) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(198607)37:4<222::aid-asi8>3.0.co;2-k · Cited by 17
What do we really know about online searchers?
Bellardo, T. (1985) · Online Review · doi:10.1108/eb024184 · Cited by 13
Planning for the information age: a survey of microcomputer use in a faculty of health sciences.
Haynes, R. B., McKibbon, K. A., & Walker, C. J. (1987) · PubMed · Cited by 11
Self-citation
Research works printed but not linked: 1
- White, 1988, Putting end-users to work, Library J., 5, 41
Cited by
Bramer, W. M., Rethlefsen, M. L., Kleijnen, J., & Franco, O. H. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0644-y · Cited by 2,114
Lowe, H. J. (1994) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1994.03510380059038 · Cited by 468
Instruments for Evaluating Education in Evidence-Based Practice: A Systematic Review
Shaneyfelt, T., Baum, K. D., Bell, D., Feldstein, D., Houston, T. K., Kaatz, S., Whelan, C., & Green, M. (2006) · JAMA · doi:10.1001/jama.296.9.1116 · Cited by 427
Rethlefsen, M. L., Farrell, A. M., Osterhaus Trzasko, L. C., & Brigham, T. J. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.11.025 · Cited by 331
The Informationist: A New Health Profession?
Davidoff, F., & Florance, V. (2000) · Annals of Internal Medicine · doi:10.7326/0003-4819-132-12-200006200-00012 · Cited by 279
Metadata
- Authors
- McKibbon, K. A., Haynes, R. B., Walker Dilks, C. J., Ramsden, M. F., Ryan, N. C., Baker, L., Flemming, T., & Fitzgerald, D.
- Affiliations
- McMaster University (CA)
- Venue
- Computers and Biomedical Research
- Publisher
- Elsevier BV
- Published
- 1990-12
- Volume
- 23
- Issue
- 6
- Pages
- 583-593
- ISSN
- 0010-4809 (ISSN-L)
- OpenAlex
- W2161479801
- Cited by
- 108
- References
- 14
- Citations by year
- 2025: 1 · 2024: 2 · 2022: 1 · 2020: 1 · 2018: 1 · 2017: 4 · 2015: 2 · 2014: 1 · 2013: 6 · 2012: 5
- FWCI
- 17.38
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- Locations
- Computers and Biomedical Research (publishedVersion)
- PubMed (publishedVersion)
- Funders
- National Institutes of Health (5r01lm004696-05)
- U.S. National Library of Medicine (R01 LM 04696-01)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
- Social Sciences › Library and Information Sciences › Library Science and Information Literacy
- Keywords
- MEDLINE searching, search quality, precision and recall, Grateful Med, physician training, information retrieval, learning curve, citation retrieval
- MeSH terms
- Evaluation Studies as Topic, Subject Headings, User-Computer Interface, Reproducibility of Results, MEDLINE, Grateful Med
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: How good are clinical MEDLINE searches? A comparative study of clinical end-user and librarian searches. https://app.cochat.ai/papers/10.1016/0010-4809(90)90042-b
- The paper
- McKibbon, K. A., Haynes, R. B., Walker Dilks, C. J., Ramsden, M. F., Ryan, N. C., Baker, L., Flemming, T., & Fitzgerald, D. (1990). How good are clinical MEDLINE searches? A comparative study of clinical end-user and librarian searches. Computers and Biomedical Research. https://doi.org/10.1016/0010-4809(90)90042-b
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.