CoChat
Sign Up
Cited by 45Meta-analysis and systematic reviewsInformation Retrieval and Search BehaviorComputational and Text Analysis Methods

Feature Engineering and a Proposed Decision-Support System for Systematic Reviewers of Medical Evidence

Open access
Bekhuis, Tanja, Tseytlin, Eugene, Mitchell, Kevin J., Demner-Fushman, Dina·2014·PLoS ONE
DOI 10.1371/journal.pone.0086277PMC PMC3903545
Open this paper in CoChat — free account

Abstract

OBJECTIVES: Evidence-based medicine depends on the timely synthesis of research findings. An important source of synthesized evidence resides in systematic reviews. However, a bottleneck in review production involves dual screening of citations with titles and abstracts to find eligible studies. For this research, we tested the effect of various kinds of textual information (features) on performance of a machine learning classifier. Based on our findings, we propose an automated system to reduce screeing burden, as well as offer quality assurance. METHODS: We built a database of citations from 5 systematic reviews that varied with respect to domain, topic, and sponsor. Consensus judgments regarding eligibility were inferred from published reports. We extracted 5 feature sets from citations: alphabetic, alphanumeric(+), indexing, features mapped to concepts in systematic reviews, and topic models. To simulate a two-person team, we divided the data into random halves. We optimized the parameters of a Bayesian classifier, then trained and tested models on alternate data halves. Overall, we conducted 50 independent tests. RESULTS: All tests of summary performance (mean F3) surpassed the corresponding baseline, P<0.0001. The ranks for mean F3, precision, and classification error were statistically different across feature sets averaged over reviews; P-values for Friedman's test were .045, .002, and .002, respectively. Differences in ranks for mean recall were not statistically significant. Alphanumeric(+) features were associated with best performance; mean reduction in screening burden for this feature type ranged from 88% to 98% for the second pass through citations and from 38% to 48% overall. CONCLUSIONS: A computer-assisted, decision support system based on our methods could substantially reduce the burden of screening citations for systematic review teams and solo reviewers. Additionally, such a system could deliver quality assurance both by confirming concordant decisions and by naming studies associated with discordant decisions for further consideration.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1371/journal.pone.0086277

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 52 · Cited by 45

Research works printed but not linked: 41

  1. [2]Luce BR, Drummond M, Jonsson B, Neumann PJ, Schwartz JS, et al (2010) EBM, HTA, and CER: clearing the confusion. Milbank Q 88: 256– 276. 20579285 PMC2980346
  2. [5]Bastian H, Glasziou P, Chalmers I (2010) Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med 7 (9) e1000326. 20877712 PMC2943439
  3. [7]Higgins JPT, Green S (2008) Cochrane Handbook for Systematic Reviews of Interventions. Chichester, UK: Wiley.
  4. [8]Institute of Medicine of the National Academies (March 2011) Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC. 24983062
  5. [9]Patient-Centered Outcomes Research Institute (December 2012) PCORI Methodology Standards. Washington, DC. pp. 1–16.
  6. [10]Bekhuis T, Demner-Fushman D (2012) Screening nonrandomized studies for medical systematic reviews: a comparative study of classifiers. Artif Intell Med 55: 197– 207. 22677493 PMC3393813
  7. [11]Reeves B, Deeks J, Higgins J, Wells G (2008) Chapter 13: Including Non-randomized Studies. In: Higgins J, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions. Chichester (UK): Wiley.
  8. [12]Cohen AM, Ambert K, McDonagh M (2009) Cross-topic learning for work prioritization in systematic review creation and update. J Am Med Inform Assoc 16: 690– 704. 19567792 PMC2744720
  9. [13]Dalal SR, Shekelle PG, Hempel S, Newberry SJ, Motala A, et al (2013) A pilot study using machine learning and domain knowledge to facilitate comparative effectiveness review updating. Med Decis Making 33: 343– 355. 22961102
  10. [14]Cohen AM (2008) Optimizing feature representation for automated systematic review work prioritization. AMIA Annu Symp Proc PMC2656096 18998798
  11. [15]Matwin S, Kouznetsov A, Inkpen D, Frunza O (2010) A new algorithm for reducing the workload of experts in performing systematic reviews. J Am Med Inform Assoc 17: 446– 453. 20595313 PMC2995653
  12. [18]Frunza O, Inkpen D, Matwin S, Klement W, O'Blenis P (2011) Exploiting the systematic review protocol for classification of medical abstracts. Artif Intell Med 51: 17– 25. 21084178
  13. [19]Cohen AM, Adams CE, Davis JM, Yu C, Yu PS, et al.. (2010) Evidence-based Medicine, the Essential Role of Systematic Reviews, and the Need for Automated Text Mining Tools. IHI'10. Arlington, Virginia, USA: ACM (978-1-4503-0030-8/10/11). pp. 376–380.
  14. [20]Yaffe J, Montgomery P, Hopewell S, Shepard LD (2012) Empty reviews: a description and consideration of Cochrane systematic reviews with no included studies. PLoS ONE 7: e36626. 22574201 PMC3344923
  15. [21]Bekhuis T, Stein CD (2013) Feasibility scans could improve prioritization and selection of systematic review topics. 21st Cochrane Colloquium. Quebec, Canada. Available: http://colloquium.cochrane.org/abstracts/feasibility-scans-could-improve-prioritization-and-selection-systematic-review-topics. Accessed 19 November 2013.
  16. [22]Abba K, Deeks JJ, Olliaro PL, Naing CM, Jackson SM, et al.. (2011) Rapid diagnostic tests for diagnosing uncomplicated P. falciparum malaria in endemic countries. Cochrane Database of Systematic Reviews Issue 7. Art. No.: CD008122. doi:10.1002/14651858.CD008122.pub2. PMC6532563 21735422 · DOI
  17. [23]Leeflang MM, Debets-Ossenkopp YJ, Visser CE, Scholten RJPM, Hooft L, et al.. (2008) Galactomannan detection for invasive aspergillosis in immunocompromized patients. Cochrane Database of Systematic Reviews Issue 4. Art. No.: CD007394. doi:10.1002/14651858.CD007394. 18843747 · DOI
  18. [24]Bekhuis T, Thyvalikakath TP, Oliver R (2009) Interventions for treating ameloblastomas of the jaws [protocol]. Cochrane Database of Systematic Reviews doi:101002/14651858CD003975pub2. Chichester, UK: John Wiley & Sons, Ltd.
  19. [25]Oremus M, Zeidler J, Ensom MHH, Matsuda-Abedini M, Balion C, et al. (Feb 2008) Utility of monitoring mycophenolic acid in solid organ transplant patients. Evidence Reports/Technology Assessments, No 164 AHRQ Publication No 08-E006. Rockville, MD: US Agency for Healthcare Research and Quality. Available: http://www.ncbi.nlm.nih.gov/books/NBK38475/. Accessed 24 July 2013.
  20. [26]Jefferson T, Di Pietrantonj C, Al-Ansary LA, Ferroni E, Thorning S, et al.. (2010) Vaccines for preventing influenza in the elderly [updated review]. Cochrane Database of Systematic Reviews Issue 2 doi:101002/14651858CD004876pub3. Chichester, UK: Wiley. 20166072
  21. [27]National Center for Biotechnology Information PubMed.gov: US National Library of Medicine, National Institutes of Health. Available: http://www.ncbi.nlm.nih.gov/pubmed/. Accessed 27 May 2013
  22. [31]Elsevier BV Indexing/Emtree. Available: http://www.elsevier.com/online-tools/embase/emtree. Accessed 29 May 2013.
  23. [32]Huang X, Lin J, Demner-Fushman D (2006) Evaluation of PICO as a knowledge representation for clinical questions. AMIA Annu Symp Proc 359– 363. 17238363 PMC1839740
  24. [34]US National Library of Medicine (2009) UMLS Reference Manual. Bethesda, MD: US National Library of Medicine. Available: http://www.ncbi.nlm.nih.gov/books/NBK9676/. Accessed 24 July 2013.
  25. [35]Zou Q, Chu WW, Morioka C, Leazer GH, Kangarloo H (2003) IndexFinder: a method of extracting key concepts from clinical texts for indexing. AMIA Annu Symp Proc 763– 767. 14728276 PMC1480259
  26. [36]Bekhuis T, Demner-Fushman D, Crowley RS (2013) Comparative effectiveness research designs: an analysis of terms and coverage in Medical Subject Headings (MeSH) and Emtree. J Med Libr Assoc 101: 92– 100. 23646024 PMC3634392
  27. [38]Blei DM, Ng AY, Jordan MI (2003) Latent Dirichlet allocation. The Journal of Machine Learning Research 3: 993– 1022.
  28. [41]Griffiths TL, Steyvers M (2004) Finding scientific topics. Proceedings of the National Academy of Sciences of the United States of America 101 Suppl 1: 5228– 5235. 14872004 PMC387300
  29. [43]Manning CD, Raghavan P, Schutze H (2008) Introduction to Information Retrieval. New York: Cambridge University Press.
  30. [44]Mierswa I, Wurst M, Klinkenberg R, Scholz M, Euler T (2006) YALE (now RapidMiner): rapid prototyping for complex data mining tasks. Proc ACM SIGKDD Int Conf on Knowl Discov Data Mining
  31. [47]Rennie J, Shih L, Teevan J, Karger D (2003) Tackling the poor assumptions of naive Bayes text classifiers. Proceedings of the Twentieth International Conference on Machine Learning (ICML). Washington, DC
  32. [49]Bekhuis T, Demner-Fushman D (2010) Towards automating the initial screening phase of a systematic review. Stud Health Technol Inform 160: 146– 150. 20841667
  33. [50]Wilczynski NL, Morgan D, Haynes RB (2005) An overview of the design and methods for retrieving high-quality studies for clinical care. BMC Med Inform Decis Mak 5: 20. 15969765 PMC1183213
  34. [51]Wilczynski NL, McKibbon KA, Walter SD, Garg AX, Haynes RB (2013) MEDLINE clinical queries are robust when searching in recent publishing years. J Am Med Inform Assoc 20: 363– 368. 23019242 PMC3638187
  35. [52]McKibbon KA, Lokker C, Handler SM, Dolovich LR, Holbrook AM, et al (2012) The effectiveness of integrated health information technologies across the phases of medication management: a systematic review of randomized controlled trials. J Am Med Inform Assoc 19: 22– 30. 21852412 PMC3240758
  36. [53]Kilicoglu H, Demner-Fushman D, Rindflesch TC, Wilczynski NL, Haynes RB (2009) Towards automatic recognition of scientifically rigorous clinical research evidence. J Am Med Inform Assoc 16: 25– 31. 18952929 PMC2605595
  37. [55]Zeng QT, Redd D, Rindflesch T, Nebeker J (2012) Synonym, topic model and predicate-based query expansion for retrieving clinical documents. AMIA Annu Symp Proc 2012: 1050– 1059. 23304381 PMC3540443
  38. [56]Jimeno-Yepes A, Wilkowski B, Mork JG, Van Lenten E, Fushman DD, et al (2011) A bottom-up approach to MEDLINE indexing recommendations. AMIA Annu Symp Proc: American Medical Informatics Association 1583. PMC3243198 22195224
  39. [59]Del Fiol G, Haug PJ (2008) Infobuttons and classification models: a method for the automatic selection of on-line information resources to fulfill clinicians' information needs. J Biomed Inform 41: 655– 666. 18249041 PMC2644726
  40. [60]Klinkenberg R, Joachims T (2000) Detecting concept drift with support vector machines. Proceedings of the Seventeenth International Conference on Machine Learning (ICML): Morgan Kaufmann 487– 494.
  41. [61]Wilczynski NL, McKibbon KA, Haynes RB (2011) Search filter precision can be improved by NOTing out irrelevant content. AMIA Annu Symp Proc 2011: 1506– 1513. 22195215 PMC3243169

Authors
Bekhuis, T., Tseytlin, E., Mitchell, K. J., & Demner-Fushman, D.
Affiliations
University of Pittsburgh (US)
Corresponding author
Bekhuis, Tanja
ORCID
Venue
PLoS ONE
Publisher
Public Library of Science (PLoS)
Published
2014-01-27
Volume
9
Issue
1
Pages
e86277
ISSN
1932-6203 (ISSN-L)
DOI
10.1371/journal.pone.0086277
PMC
PMC3903545
OpenAlex
W2079370125
Cited by
45
References
52
Citations by year
2026: 1 · 2025: 1 · 2024: 3 · 2023: 5 · 2022: 1 · 2021: 4 · 2020: 6 · 2019: 4 · 2018: 3 · 2017: 5 · 2016: 4 · 2015: 7 · 2014: 1
FWCI
1.55
Type
journal article
Language
en
Open access
Licence
Locations
  • PLoS ONE (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • D-Scholarship@Pitt (University of Pittsburgh) (acceptedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • National Institutes of Health
  • U.S. National Library of Medicine (R00LM010943)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
Keywords
systematic reviews, citation screening, feature engineering, decision support system, topic modeling, Bayesian classifier, quality assurance, evidence-based medicine
MeSH terms
Artificial Intelligence, Bayes Theorem, Decision Support Techniques, Publications, Review Literature as Topic, Databases, Bibliographic, Evidence-Based Medicine

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review