Automatically finding relevant citations for clinical guideline development
Open accessThe DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Abstract
OBJECTIVE: Literature database search is a crucial step in the development of clinical practice guidelines and systematic reviews. In the age of information technology, the process of literature search is still conducted manually, therefore it is costly, slow and subject to human errors. In this research, we sought to improve the traditional search approach using innovative query expansion and citation ranking approaches. METHODS: We developed a citation retrieval system composed of query expansion and citation ranking methods. The methods are unsupervised and easily integrated over the PubMed search engine. To validate the system, we developed a gold standard consisting of citations that were systematically searched and screened to support the development of cardiovascular clinical practice guidelines. The expansion and ranking methods were evaluated separately and compared with baseline approaches. RESULTS: Compared with the baseline PubMed expansion, the query expansion algorithm improved recall (80.2% vs. 51.5%) with small loss on precision (0.4% vs. 0.6%). The algorithm could find all citations used to support a larger number of guideline recommendations than the baseline approach (64.5% vs. 37.2%, p<0.001). In addition, the citation ranking approach performed better than PubMed's "most recent" ranking (average precision +6.5%, recall@k +21.1%, p<0.001), PubMed's rank by "relevance" (average precision +6.1%, recall@k +14.8%, p<0.001), and the machine learning classifier that identifies scientifically sound studies from MEDLINE citations (average precision +4.9%, recall@k +4.2%, p<0.001). CONCLUSIONS: Our unsupervised query expansion and ranking techniques are more flexible and effective than PubMed's default search engine behavior and the machine learning classifier. Automated citation finding is promising to augment the traditional literature search.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Fihn, S. D., Gardin, J. M., Abrams, J., Berra, K., Blankenship, J. C., Dallas, A. P., Douglas, P. S., Foody, J. M., Gerber, T. C., Hinderliter, A. L., et al. (2012) · Circulation · doi:10.1161/cir.0b013e318277d6a0 · Cited by 845
Reducing Workload in Systematic Review Preparation Using Automated Citation Classification
Cohen, A. M., Hersh, W. R., Peterson, K., & Yen, P.-Y. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1929 · Cited by 411 · cited 1×
“investigated a machine learning approach to solve a classification problem in drug effectiveness reviews [12,17].”
BackgroundThe information‐seeking behaviour of doctors: a review of the evidence
Davies, K. (2007) · Health Information & Libraries Journal · doi:10.1111/j.1471-1842.2007.00713.x · Cited by 356 · cited 1×
“In a review of information-seeking behavior, Davies showed that clinicians’ lack of time, issues with information technology, limited search skills are top barriers for information searching [5].”
BackgroundSemi-automated screening of biomedical citations for systematic reviews
Wallace, B. C., Trikalinos, T. A., Lau, J., Brodley, C., & Schmid, C. H. (2010) · BMC Bioinformatics · doi:10.1186/1471-2105-11-55 · Cited by 351
Chiang, C.-E., Naditch-Brûlé, L., Murin, J., Goethals, M., Inoue, H., O’Neill, J., Silva-Cardoso, J., Zharinov, O., Gamra, H., Alam, S., et al. (2012) · Circulation: Arrhythmia and Electrophysiology · doi:10.1161/circep.112.970749 · Cited by 327
Esmolol versus verapamil in the acute treatment of atrial fibrillation or atrial flutter
Platia, E. V., Michelson, E. L., Porterfield, J. K., & Das, G. (1989) · The American Journal of Cardiology · doi:10.1016/0002-9149(89)90141-0 · Cited by 213
Ellenbogen, K. A., Dias, V. C., Plumb, V. J., Heywood, J. T., & Mirvis, D. M. (1991) · Journal of the American College of Cardiology · doi:10.1016/0735-1097(91)90743-s · Cited by 160
Towards Automatic Recognition of Scientifically Rigorous Clinical Research Evidence
Kilicoglu, H., Demner-Fushman, D., Rindflesch, T. C., Wilczynski, N. L., & Haynes, R. B. (2009) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2996 · Cited by 101 · cited 1×
“implemented an ensemble approach combining several machine learning classifiers (Naïve Bayes, support vector machine (SVM), and boosting) to identify scientifically rigorous studies [45].”
MethodSiu, C.-W., Lau, C.-P., Lee, W.-L., Lam, K.-F., & Tse, H.-F. (2009) · Critical Care Medicine · doi:10.1097/ccm.0b013e3181a02f56 · Cited by 96
Cross-Topic Learning for Work Prioritization in Systematic Review Creation and Update
Cohen, A. M., Ambert, K., & McDonagh, M. (2009) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m3162 · Cited by 88 · cited 1×
“Informatics research has investigated machine learning approaches to prioritize citation screening in systematic reviews [14,22,23].”
Background
Cited by
Automation of systematic literature reviews: A systematic literature review
van Dinter, R., Tekinerdogan, B., & Catal, C. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106589 · Cited by 432
Bui, D. D. A., Del Fiol, G., Hurdle, J. F., & Jonnalagadda, S. (2016) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2016.10.014 · Cited by 70
“Mapping text to UMLS concepts is one of the extraction methods commonly used in clinical and biomedical NLP studies (38-40).”
MethodSelf-citationPDF text classification to leverage information extraction from publication reports
Bui, D. D. A., Del Fiol, G., & Jonnalagadda, S. (2016) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2016.03.026 · Cited by 52
“The SR process involves a series of scientifically rigorous steps(8), such as citation searching(9), abstract screening, full-text screening, data extraction, and article appraisal.”
MethodSelf-citationSemi-automated Tools for Systematic Searches
Adam, G. P., Wallace, B. C., & Trikalinos, T. A. (2022) · Methods in Molecular Biology · doi:10.1007/978-1-0716-1566-9_2 · Cited by 19
Metadata
- Authors
- Bui, D. D. A., Jonnalagadda, S., & Del Fiol, G.
- Venue
- Journal of Biomedical Informatics
- Publisher
- Elsevier BV
- Published
- 2015-10
- Volume
- 57
- Pages
- 436-445
- ISSN
- 1532-0464 (ISSN-L)
- OpenAlex
- W2163168379
- Cited by
- 19
- References
- 10
- FWCI
- 2.70
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Automatically finding relevant citations for clinical guideline development. https://app.cochat.ai/papers/10.1016/j.jbi.2015.09.003
- The paper
- Bui, D. D. A., Jonnalagadda, S., & Del Fiol, G. (2015). Automatically finding relevant citations for clinical guideline development. Journal of Biomedical Informatics. https://doi.org/10.1016/j.jbi.2015.09.003
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.