CoChat
Sign Up
Cited by 31Biomedical Text Mining and OntologiesMeta-analysis and systematic reviewsAdvanced Text Analysis Techniques

Evaluation of a rule-based method for epidemiological document classification towards the automation of systematic reviews

Open access
Karystianis, George, Thayer, Kristina, Wolfe, Mary, Tsafnat, Guy·2017·Journal of Biomedical Informatics
DOI 10.1016/j.jbi.2017.04.004
Open this paper in CoChat — free accountOpen PDF

Abstract

Introduction Most data extraction efforts in epidemiology are focused on obtaining targeted information from clinical trials. In contrast, limited research has been conducted on the identification of information from observational studies, a major source for human evidence in many fields, including environmental health. The recognition of key epidemiological information (e.g., exposures) through text mining techniques can assist in the automation of systematic reviews and other evidence summaries. Method We designed and applied a knowledge-driven, rule-based approach to identify targeted information (study design, participant population, exposure, outcome, confounding factors, and the country where the study was conducted) from abstracts of epidemiological studies included in several systematic reviews of environmental health exposures. The rules were based on common syntactical patterns observed in text and are thus not specific to any systematic review. To validate the general applicability of our approach, we compared the data extracted using our approach versus hand curation for 35 epidemiological study abstracts manually selected for inclusion in two systematic reviews. Results The returned F-score, precision, and recall ranged from 70% to 98%, 81% to 100%, and 54% to 97%, respectively. The highest precision was observed for exposure, outcome and population (100%) while recall was best for exposure and study design with 97% and 89%, respectively. The lowest recall was observed for the population (54%), which also had the lowest F-score (70%). Conclusion The generated performance of our text-mining approach demonstrated encouraging results for the identification of targeted information from observational epidemiological study abstracts related to environmental exposures. We have demonstrated that rules based on generic syntactic patterns in one corpus can be applied to other observational study design by simple interchanging the dictionaries aiming to identify certain characteristics (i.e., outcomes, exposures). At the document level, the recognised information can assist in the selection and categorization of studies included in a systematic review.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.jbi.2017.04.004

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The 'Other' category includes some more detailed drug annotations or information such as confounders and other entity types.

    “The ‘Other’ category includes some more detailed drug annotations 65 or information such as confounders 49 and other entity types (see the full dataset in Underlying data for more information 127 ).”

    From Data extraction methods for systematic review (semi)automation: a living systematic review (2021) · cites this paper as [49]

  • Scores might not be similar between different datasets, especially for rule-based classifiers, when datasets are small, when structure and distribution of class of interest varies, or when annotation guidelines vary.

    “Therefore, scores might not be similar between different datasets, especially for rule-based classifiers, 80 when datasets are small, 35 , 49 when structure and distribution of class of interest varies, 40 or when the annotation guidelines vary. 85”

    From Data extraction methods for systematic review (semi)automation: a living systematic review (2021) · cites this paper as [49]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 32

  1. Glasziou, 2004, Assessing the quality of research, BMJ: Br. Med. J., 328, 39, 10.1136/bmj.328.7430.39 · DOI · Paper page
  2. Tooth, 2005, Quality of reporting of observational longitudinal research, Am. J. Epidemiol., 161, 280, 10.1093/aje/kwi042 · DOI · Paper page
  3. Pocock, 2004, Issues in the reporting of epidemiological studies: a survey of recent practice, BMJ, 329, 883, 10.1136/bmj.38250.571088.55 · DOI · Paper page
  4. Jonnalagadda, 2015, Automating data extraction in systematic reviews: a systematic review, System. Rev., 4, 78, 10.1186/s13643-015-0066-7 · DOI · Paper page
  5. Korhonen, 2012, Text mining for literature review and knowledge discovery in cancer risk assessment and research, PLoS ONE, 7, e33427, 10.1371/journal.pone.0033427 · DOI
  6. Chiang, 2010, Automated evaluation of electronic discharge notes to assess quality of care for cardiovascular diseases using Medical Language Extraction and Encoding System (MedLEE), J. Am. Med. Inform. Assoc., 17, 245, 10.1136/jamia.2009.000182 · DOI
  7. Fiszman, 2000, Automatic detection of acute bacterial pneumonia from chest X-ray reports, J. Am. Med. Inform. Assoc., 7, 593, 10.1136/jamia.2000.0070593 · DOI · Paper page
  8. Erhardt, 2006, Status of text-mining techniques applied to biomedical text, Drug Discovery Today, 11, 315, 10.1016/j.drudis.2006.02.011 · DOI
  9. Rodriguez-Esteban, 2009, Biomedical text mining and its applications, PLoS Comput. Biol., 5, e1000597, 10.1371/journal.pcbi.1000597 · DOI
  10. B. De Bruijn, S. Carini, S. Kiritchenko, J. Martin, I. Sim (Eds.), Automated information extraction of key trial design elements from clinical trial publications, in: AMIA Annual Symposium Proceedings; 2008: American Medical Informatics Association.
  11. M.J. Hansen, N. Rasmussen, G. Chung (Eds.), Extracting number of trial participants from abstracts of randomized controlled trials, in: Proceedings of Tromsø Telemedicine and eHealth Conference: 9–11 June 2008; Tromsø, Norway; 2008.
  12. Hara, 2007, Extracting clinical trial design information from MEDLINE abstracts, New Generation Comput., 25, 263, 10.1007/s00354-007-0017-5 · DOI · Paper page
  13. R. Xu, Y. Garten, K.S. Supekar, A.K. Das, R.B. Altman, A.M. Garber (Eds.), Extracting subject demographic information from abstracts of randomized clinical trial reports, in: Medinfo 2007: Proceedings of the 12th World Congress on Health (Medical) Informatics; Building Sustainable Health Systems, IOS Press, 2007.
  14. Z. Luo, S.B. Johnson, A.M. Lai, C. Weng (Eds.), Extracting temporal constraints from clinical research eligibility criteria using conditional random fields, in: AMIA Annu Symp Proc, 2011.
  15. Luo, 2013, A human–computer collaborative approach to identifying common data elements in clinical trial eligibility criteria, J. Biomed. Inform., 46, 33, 10.1016/j.jbi.2012.07.006 · DOI
  16. Kiritchenko, 2010, ExaCT: automatic extraction of clinical trial characteristics from journal publications, BMC Med. Inform. Decis. Mak., 10, 56, 10.1186/1472-6947-10-56 · DOI · Paper page
  17. Zeng, 2006, Extracting principal diagnosis, co-morbidity and smoking status for asthma research: evaluation of a natural language processing system, BMC Med. Inform. Decis. Mak., 6, 30, 10.1186/1472-6947-6-30 · DOI · Paper page
  18. Y. Wang (Ed.), Annotating and recognising named entities in clinical notes, in: Proceedings of the ACL-IJCNLP 2009 Student Research Workshop, Association for Computational Linguistics, 2009. · DOI
  19. Shatkay, 2003, Mining the biomedical literature in the genomic era: an overview, J. Comput. Biol., 10, 821, 10.1089/106652703322756104 · DOI
  20. Zhu, 2013, Biomedical text mining and its applications in cancer research, J. Biomed. Inform., 46, 200, 10.1016/j.jbi.2012.10.007 · DOI
  21. Spasić, 2014, Text mining of cancer-related information: review of current status and future directions, Int. J. Med. Informatics, 83, 605, 10.1016/j.ijmedinf.2014.06.009 · DOI
  22. Coden, 2009, Automatically extracting cancer disease characteristics from pathology reports into a Disease Knowledge Representation Model, J. Biomed. Inform., 42, 937, 10.1016/j.jbi.2008.12.005 · DOI
  23. Zweigenbaum, 2007, Frontiers of biomedical text mining: current progress, Brief. Bioinform., 8, 358, 10.1093/bib/bbm045 · DOI
  24. G.B. Hamra, N. Guha, A. Cohen, F. Laden, O. Raaschou-Nielsen, J.M. Samet, et al., Outdoor particulate matter exposure and lung cancer: a systematic review and meta-analysis, 2014. · DOI
  25. Zhao, 2015, The correlation between Polybrominated Diphenyl Ethers (PBDEs) and thyroid hormones in the general population: a meta-analysis, PLoS ONE, 10, e0126989, 10.1371/journal.pone.0126989 · DOI
  26. K. Thayer, A. Rooney, A. Boyles, S. Holmgren, V. Walker, G. Kissling, et al., Draft protocol for systematic review to evaluate the evidence for an association between bisphenol A (BPA) exposure and obesity, National Toxicology Program, 2013.
  27. NTP Monograph: Identifying Research Needs for Assessing Safe Use of High Intakes of Folic Acid, 2015, pp. 1–51.
  28. Cunningham, 2013, Getting more out of biomedical documents with GATE's full lifecycle open source text analytics, PLoS Comput. Biol., 9, e1002854, 10.1371/journal.pcbi.1002854 · DOI
  29. Ananiadou, 2006, Text mining and its potential applications in systems biology, Trends Biotechnol., 24, 571, 10.1016/j.tibtech.2006.10.002 · DOI
  30. L. Bero, G. Busuttil, C. Farquhar, T.P. Koehlmoos, D. Moher, M. Nylenna, et al., Measuring the performance of the Cochrane library, The Cochrane database of systematic reviews, 2012;12:ED000048. · DOI
  31. Fiszman, 2007, Identifying Risk Factors for Metabolic Syndrome in Biomedical Text, AMIA Annu. Symp. Proc., 249
  32. H. Xu, Y. Lu, M. Jiang, M. Liu, J.C. Denny, Q. Dai, et al. (Eds.), Mining biomedical literature for terms related to epidemiologic exposures, in: AMIA Annual Symposium Proceedings: American Medical Informatics Association, 2010.

Cited by

Sorted by citations

Authors
Karystianis, G., Thayer, K., Wolfe, M., & Tsafnat, G.
Affiliations
  • Macquarie University (AU)
  • National Institute of Environmental Health Sciences (US)
Corresponding author
Karystianis, George
ORCID
Venue
Journal of Biomedical Informatics
Publisher
Elsevier BV
Published
2017-06
Volume
70
Pages
27-34
ISSN
1532-0464 (ISSN-L)
DOI
10.1016/j.jbi.2017.04.004
OpenAlex
W2609740230
Cited by
31
References
5
Citations by year
2026: 3 · 2025: 2 · 2024: 10 · 2023: 4 · 2022: 3 · 2021: 3 · 2019: 4 · 2018: 2
FWCI
0.16
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of Biomedical Informatics (publishedVersion)
  • PubMed (publishedVersion)
Topics
  • Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
Keywords
document classification, data extraction, systematic reviews, observational studies, outcome identification, syntactic patterns
MeSH terms
Automation, Review Literature as Topic, Data Mining

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review