CoChat
Sign Up
Cited by 113Biomedical Text Mining and OntologiesMachine Learning in HealthcarePharmacovigilance and Adverse Drug Reactions

Adverse drug events and medication relation extraction in electronic health records with ensemble deep learning methods

Open access
Christopoulou, Fenia, Tran, Thy Thy, Sahu, Sunil Kumar, Miwa, Makoto, Ananiadou, Sophia·2020·Journal of the American Medical Informatics Association
DOI 10.1093/jamia/ocz101PMC PMC6913215
Open this paper in CoChat — free accountOpen PDF

Abstract

Objective: Identification of drugs, associated medication entities, and interactions among them are crucial to prevent unwanted effects of drug therapy, known as adverse drug events. This article describes our participation to the n2c2 shared-task in extracting relations between medication-related entities in electronic health records. Materials and Methods: We proposed an ensemble approach for relation extraction and classification between drugs and medication-related entities. We incorporated state-of-the-art named-entity recognition (NER) models based on bidirectional long short-term memory (BiLSTM) networks and conditional random fields (CRF) for end-to-end extraction. We additionally developed separate models for intra- and inter-sentence relation extraction and combined them using an ensemble method. The intra-sentence models rely on bidirectional long short-term memory networks and attention mechanisms and are able to capture dependencies between multiple related pairs in the same sentence. For the inter-sentence relations, we adopted a neural architecture that utilizes the Transformer network to improve performance in longer sequences. Results: Our team ranked third with a micro-averaged F1 score of 94.72% and 87.65% for relation and end-to-end relation extraction, respectively (Tracks 2 and 3). Our ensemble effectively takes advantages from our proposed models. Analysis of the reported results indicated that our proposed approach is more generalizable than the top-performing system, which employs additional training data- and corpus-driven processing techniques. Conclusions: We proposed a relation extraction system to identify relations between drugs and medication-related entities. The proposed approach is independent of external syntactic tools. Analysis showed that by using latent Drug-Drug interactions we were able to significantly improve the performance of non–Drug-Drug pairs in EHRs.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/jamia/ocz101

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Previous studies have shown that adopting an ensemble method could further improve clinical NER performances.

    “Previous studies [38-41] have shown that adopting an ensemble method could further improve the clinical NER performances.”

    From Extracting Family History of Patients From Clinical Narratives: Exploring an End-to-End Solution With Deep Learning Models (2020) · cites this paper as [41]

    The Experimental design section says the authors selected an NER ensemble with the highest F1-score on the development set. However, the Results report improvements for relation extraction and the end-to-end pipeline, not a comparison showing that ensembling improved NER performance over a non-ensemble model. Page numbers are unavailable in our copy. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

References 22 · Cited by 113

Research works printed but not linked: 42

  1. [1]Bates DW, Cullen DJ, Laird N. Incidence of adverse drug events and potential adverse drug events: implications for prevention. JAMA 1995; 274 (1): 29– 34. 7791255
  2. [2]Wang X, Hripcsak G, Markatou M, et al Active computerized pharmacovigilance using natural language processing, statistics, and electronic health records: a feasibility study. J Am Med Inform Assoc 2009; 16 (3): 328– 37. 19261932 PMC2732239
  3. [3]Kang N, Singh B, Bui C, et al Knowledge-based extraction of adverse drug events from biomedical text. BMC Bioinformatics 2014; 15 (1): 64. 24593054 PMC3973995
  4. [4]Dasgupta T, Naskar A, Dey L. Exploring linguistic and graph based features for the automatic classification and extraction of adverse drug effects. In: Proceedings of CICLing; 2017: 463– 74.
  5. [5]Uzuner Ö, South BR, Shen S, et al 2010 i2b2/VA challenge on concepts, assertions, and relations in clinical text. J Am Med Inform Assoc 2011; 18 (5): 552– 6. 21685143 PMC3168320
  6. [6]Roberts K, Rink B, Harabagiu S. Extraction of medical concepts, assertions, and relations from discharge summaries for the fourth i2b2/VA shared task. In: Proceedings of i2b2/VA Workshop on Challenges in Natural Language Processing for Clinical Data; 2010.
  7. [7]de Bruijn B, Cherry C, Kiritchenko S, et al NRC at i2b2: one challenge, three practical tasks, nine statistical systems, hundreds of clinical records, millions of useful features. In: Proceedings of i2b2/VA Workshop on Challenges in Natural Language Processing for Clinical Data; 2010.
  8. [8]Sahu SK, Anand A, Oruganty K, et al Relation extraction from clinical texts using domain invariant convolutional neural network. In: Proceedings of BioNLP; 2016: 206-15.
  9. [9]Gurulingappa H, Rajput AM, Roberts A, et al Development of a benchmark corpus to support the automatic extraction of drug-related adverse effects from medical case reports. J Biomed Inform 2012; 45 (5): 885– 92. 22554702
  10. [10]Hochreiter S, Schmidhuber J. Long short-term memory. Neural Comput 1997; 9 (8): 1735– 80. 9377276
  11. [11]Li F, Liu W, Yu H. Extraction of information related to adverse drug events from electronic health record notes: design of an end-to-end model based on deep learning. JMIR Med Inform 2018; 6 (4): e12159. 30478023 PMC6288593
  12. [12]Ramamoorthy S, Murugan S. An attentive sequence model for adverse drug event extraction from biomedical text. arXiv 2018 Jan 2 [E-pub ahead of print].
  13. [13]Bekoulis G, Deleu J, Demeester T, Develder C. Joint entity recognition and relation extraction as a multi-head selection problem. Expert Syst Appl 2018; 114: 34– 45.
  14. [14]Bekoulis G, Deleu J, Demeester T. Adversarial training for multi-context joint entity and relation extraction. In: Proceedings of EMNLP; 2018: 2830–36.
  15. [15]Zhao J, Zhou T, Dai W. Convolutional neural network-based joint extraction of adverse drug events. In: Proceedings of ICCSE; 2018: 1– 5.
  16. [16]Roberts K, Demner-Fushman D, Tonning JM. Overview of the TAC 2017 adverse reaction extraction from drug labels track. In: Proceedings of TAC; 2017.
  17. [17]Xu J, Lee HJ, Ji Z, et al UTH_CCB system for adverse drug reaction extraction from drug labels at TAC-ADR. In: Proceedings of TAC; 2017.
  18. [18]Dandala B, Mahajan D, Devarakonda M. IBM Research System at TAC 2017: adverse drug reactions extraction from drug labels. In: Proceedings of TAC; 2017.
  19. [19]Munkhdalai T, Liu F, Yu H. Clinical relation extraction toward drug safety surveillance using electronic health record narratives: classical learning versus deep learning. JMIR Public Health Surveill 2018; 4 (2): e29. 29695376 PMC5943628
  20. [20]Jagannatha A, Liu F, Liu W, et al Overview of the first natural language processing challenge for extracting medication, indication, and adverse drug events from electronic health record notes (MADE 1.0.). Drug Saf 2018; 42 (1): 99–111. PMC6860017 30649735
  21. [21]Dandala B, Joopudi V, Devarakonda M. IBM Research System at MADE 2018: detecting adverse drug events from electronic health records. In: International Workshop on Medication and Adverse Drug Event Detection; 2018: 39– 47.
  22. [22]Li F, Zhang M, Fu G, et al A neural joint model for entity and relation extraction from biomedical text. BMC Bioinform 2018; 18 (1): 198. PMC5374588 28359255
  23. [23]Christopoulou F, Miwa M, Ananiadou S. A walk-based model on entity graphs for relation extraction. In: Proceedings of ACL; 2018(2): 81–8.
  24. [24]Yi Z, Li S, Yu J, et al Drug-drug interaction extraction via recurrent neural network with multiple attention layers. In: Proceedings of ADMA; 2017: 554– 66.
  25. [25]Björne J, Salakoski T. Biomedical event extraction using convolutional neural networks and dependency parsing. In: Proceedings of BioNLP; 2018: 98– 108.
  26. [26]Zeng D, Liu K, Lai S, et al Relation classification via convolutional deep neural network. In: Proceedings of COLING; 2014: 2335– 44.
  27. [27]Vaswani A, Shazeer N, Parmar N, et al Attention is all you need. Adv Neural Inf Process Syst 2017; 30: 5998– 6008.
  28. [28]Liu R, Diwan M, AbdulHameed MDM, et al Data-driven prediction of adverse drug reactions induced by drug-drug interactions. BMC Pharmacol Toxicol 2017; 18 (1): 44. 28595649 PMC5465578
  29. [29]Verga P, Strubell E, McCallum A. Simultaneously self-attending to all mentions for full-abstract biological relation extraction. In: Proceedings of NAACL-HLT 2018; (1): 872–84.
  30. [30]He K, Zhang X, Ren S, et al Deep residual learning for image recognition. In: Proceedings of IEEE CVPR; 2016: 770-78.
  31. [31]Ju M, Miwa M, Ananiadou S. A neural layered model for nested named entity recognition. In: Proceedings of NAACL-HLT; 2018(1): 1446–59.
  32. [32]Ju M, Nguyen NTH, Miwa M, et al An Ensemble of Neural Model for Nested Adverse Drug Events and Medication Extraction with Subwords [published online ahead of print June 14, 2019]. J Am Med Inform Assoc 2020; 27 (1): 22– 30. PMC6913208 31197355
  33. [33]Johnson AEW, Pollard TJ, Shen L, et al MIMIC-III, a freely accessible critical care database. Sci Data 2016; 3: 160035. 27219127 PMC4878278
  34. [34]Jessop DM, Adams SE, Willighagen EL, et al OSCAR4: a flexible architecture for chemical text-mining. J Cheminform 2011; 3 (1): 41. 21999457 PMC3205045
  35. [35]Chiu B, Crichton G, Korhonen A, et al How to train good word embeddings for biomedical NLP. In: Proceedings of BioNLP; 2016: 166– 174.
  36. [36]Lam L, Suen SY. Application of majority voting to pattern recognition: an analysis of its behavior and performance. Trans of SMC 1997; 27 (5): 553– 68.
  37. [37]Mu X, Lu J, Watta P, et al Weighted voting-based ensemble classifiers with application to human face recognition and voice recognition. In: Proceedings of IJCNN; 2009: 2168– 71.
  38. [38]Noreen EW. Computer-Intensive Methods for Testing Hypotheses. New York, NY: Wiley; 1989.
  39. [39]Zhou P, Shi W, Tian J, et al Attention-based bidirectional long short-term memory networks for relation classification. In: Proceedings of ACL; 2018(2): 207–12.
  40. [40]Wei Q, Ji Z, Li Z, et al A study of deep learning approaches for medication and adverse drug event extraction from clinical text. [published online ahead of print May, 2019]. J Am Med Inform Assoc 2020; 27 (1): 13– 21. PMC6913210 31135882
  41. [41]Kuhn M, Letunic I, Jensen LJ, et al The SIDER database of drugs and side effects. Nucleic Acids Res 2016; 44 (D1): D1075– 9. 26481350 PMC4702794
  42. [42]Miwa M, Bansal M. End-to-end relation extraction using lstms on sequences and tree structures. In: Proceedings of ACL; 2016(1): 1105-16.

Authors
Christopoulou, F., Tran, T. T., Sahu, S. K., Miwa, M., & Ananiadou, S.
Affiliations
  • University of Manchester (GB)
  • Artificial Intelligence Research Center (JP)
Corresponding author
Ananiadou, Sophia
ORCID
Venue
Journal of the American Medical Informatics Association
Publisher
Oxford University Press (OUP)
Published
2019-08-07
Volume
27
Issue
1
Pages
39-46
ISSN
  • 1067-5027 (ISSN-L)
  • 1527-974X
DOI
10.1093/jamia/ocz101
PMC
PMC6913215
OpenAlex
W2947596730
Cited by
113
References
22
Citations by year
2026: 12 · 2025: 12 · 2024: 14 · 2023: 16 · 2022: 23 · 2021: 23 · 2020: 11 · 2019: 2
FWCI
4.88
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of the American Medical Informatics Association (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Research Explorer (The University of Manchester) (publishedVersion, cc-by)
Funders
  • Biotechnology and Biological Services Research Council EMPATHY (BB/M006891/1)
  • Manchester Molecular Pathology Innovation Centre (MR/N00583X/1)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Life Sciences › Pharmacology, Toxicology and Pharmaceutics › Toxicology › Pharmacovigilance and Adverse Drug Reactions
Keywords
adverse drug reactions, electronic health records, ensemble deep learning, named entity recognition, drug-drug interactions, BiLSTM, conditional random fields, transformer models
MeSH terms
Deep Learning, Drug Interactions, Humans, Natural Language Processing, Information Storage and Retrieval, Neural Networks, Computer, Electronic Health Records, Drug-Related Side Effects and Adverse Reactions

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review