CoChat
Sign Up
Cited by 29Biomedical Text Mining and OntologiesMachine Learning in HealthcareTopic Modeling

Extracting Family History of Patients From Clinical Narratives: Exploring an End-to-End Solution With Deep Learning Models

Open access
Yang, Xi, Zhang, Hansi, He, Xing, Bian, Jiang, Wu, Yonghui·2020·JMIR Medical Informatics
DOI 10.2196/22982PMC PMC7772072

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

Background: Patients’ family history (FH) is a critical risk factor associated with numerous diseases. However, FH information is not well captured in the structured database but often documented in clinical narratives. Natural language processing (NLP) is the key technology to extract patients’ FH from clinical narratives. In 2019, the National NLP Clinical Challenge (n2c2) organized shared tasks to solicit NLP methods for FH information extraction. Objective: This study presents our end-to-end FH extraction system developed during the 2019 n2c2 open shared task as well as the new transformer-based models that we developed after the challenge. We seek to develop a machine learning–based solution for FH information extraction without task-specific rules created by hand. Methods: We developed deep learning–based systems for FH concept extraction and relation identification. We explored deep learning models including long short-term memory-conditional random fields and bidirectional encoder representations from transformers (BERT) as well as developed ensemble models using a majority voting strategy. To further optimize performance, we systematically compared 3 different strategies to use BERT output representations for relation identification. Results: Our system was among the top-ranked systems (3 out of 21) in the challenge. Our best system achieved micro-averaged F1 scores of 0.7944 and 0.6544 for concept extraction and relation identification, respectively. After challenge, we further explored new transformer-based models and improved the performances of both subtasks to 0.8249 and 0.6775, respectively. For relation identification, our system achieved a performance comparable to the best system (0.6810) reported in the challenge. Conclusions: This study demonstrated the feasibility of utilizing deep learning methods to extract FH information from clinical narratives.

AI snapshot

Written from the paper’s full text.

TL;DR

This paper presents an end-to-end deep learning system for extracting family history from clinical narratives in the 2019 n2c2 shared task and post-challenge transformer models. The best post-challenge system achieved micro-F1 scores of 0.8249 for concept extraction and 0.6775 for relation identification.

Key findings

  • The best system during the challenge (LSTM-CRFs-EN + BERT-cls + BERT-rel) achieved micro-F1 scores of 0.7944 for concept extraction and 0.6544 for relation identification, ranking third among 21 submitted systems from 17 teams in subtask 2.
  • After the challenge, BERT-ner-EN + BERT-cls + BERT-rel improved F1 to 0.8249 for concept extraction and 0.6775 for relation identification; the relation result was comparable to the best challenge score of 0.6810.
  • BERT-ner outperformed LSTM-CRFs for observation concepts (F1 0.8094 vs 0.7833) but was marginally lower for family member concepts (0.8066 vs 0.8069); ensembling improved family members by about 2% and observations by about 1.5%.
  • For relation categories, the best system achieved F1 scores of 0.6821 for family member–living status and 0.6760 for family member–observation; for BERT output representations, combining [CLS] and start-position entity markers gave an averaged F1 of 0.8975 versus 0.8851 and 0.8904 for the other two strategies.

Methods

  • Data: the 2019 n2c2 family history corpus of 216 clinical notes from the Mayo Clinic data warehouse, split into 99 training notes and 117 test notes, with entity-level and document-level annotations for family members, observations, living status, and relations.
  • System: a five-module pipeline (preprocessing, NER, classification, relation identification, postprocessing); NER used BIO tagging with LSTM-CRFs and BERT-LARGE, relation identification was framed as binary classification with candidate pairs restricted to family member plus nonfamily member and cross-distance less than 3, and BERT classifiers handled family role, family side, negation, and living status.
  • Evaluation: micro-averaged precision, recall, and F1 using the official n2c2 evaluation script; hyperparameters were optimized via fivefold cross-validation, and experiments used 2 NVIDIA P6000 GPUs.

Limitations

  • The paper states that limited clinical corpora for FH extraction are available because annotating clinical notes is expensive and time-consuming, and suggests data augmentation such as generative adversarial networks as a potential solution.
  • The paper states that the system is a two-stage pipeline, so errors from NER propagate to relation extraction; future work may explore joint learning to alleviate this.
  • Evident from the text: the training set is small (99 notes), and some entity types are rare (eg, “parent” appeared once and “grandparent” entities appeared 6 times), which may limit performance and generalizability.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.2196/22982

Cites

54 of 54 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

17 of 32 cited sources checked · 8 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    Family history is a critical risk factor associated with numerous diseases.

    “Patients’ family history (FH) is a critical risk factor associated with numerous diseases [1-3] such as diabetes [4], coronary heart disease [5], and multiple types of cancers [6-9].”
    Quote in the paperSubstantial claim1 of 5 sources checked
    • cites [1] source
    • cites [2] source
    • cites [3] source
    • cites [4] source
    • cites [5] source
  • Findingown data

    We screened 4 different word embeddings and found that the Common Crawl embeddings achieved better performance compared to other embeddings on a validation data set.

    “We screened 4 different word embeddings following a similar procedure reported in our previous study [46] and found that the Common Crawl embeddings—released by Facebook and trained using the fastText on the Common Crawl data set [47]—achieved better performance compared to other embeddings on a validation data set.”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites [46] source
    • cites [47] source

      The Implementation details section says the paper’s subword word-vector model is publicly available, and the copy links to facebookresearch/fastText. But the Datasets section says its models were trained chiefly on Wikipedia, not Common Crawl; the copy does not identify a Common Crawl embedding release. The validation-set comparison is presented as the citing authors’ own result, not a result attributed to [47]. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
  • Premisecitation

    There are only a handful of studies focused on extracting family history of patients.

    “However, there are only a handful of studies focused on extracting FH of patients [30-32]”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites [30] Wang, Y, 2017, AMIA Jt Summits Transl Sci Proc source

      The paper’s introduction describes GLUE as a benchmark for natural-language-understanding tasks, and its Diagnostic Dataset section discusses linguistic phenomena, not patient records. Searches for family-history extraction and clinical information extraction found no passage assessing how many studies address that topic. No page numbers are available in our copy. (LLM, conf=0.78)

      Experimental — accuracy not yet measured
    • cites [31] Goryachev, S, 2008, AMIA Annu Symp Proc

      Not checked in the free run.

    • cites [32] Bill, R, 2014, AMIA Annu Symp Proc

      Not checked in the free run.

  • Findingown data

    Our best submission during the original challenge achieved a micro-averaged F1 score of 0.6544 for relation identification (subtask 2).

    “Our best submission during the original challenge (LSTM-CRFs-EN + BERT-cls +BERT-rel) achieved F1 scores of 0.7944 and 0.6544 for subtask 1 and subtask 2, respectively”
    • Also stated in: § Results · plain
      “Our best system achieved micro-averaged F1 scores of 0.7944 and 0.6544 for concept extraction and relation identification, respectively.”
    Quote in the paperSubstantial claim
  • Findingown data

    After the challenge, the combination of BERT-ner-EN, BERT-cls, and BERT-rel achieved a micro-averaged F1 score of 0.6775 for subtask 2.

    “After the challenge, we further explored the BERT model for NER and the combination of BERT-ner-EN, BERT-cls, and BERT-rel achieved better F1 scores of 0.8249 and 0.6775 for the 2 subtasks, respectively.”
    • Also stated in: § Results · plain
      “After challenge, we further explored new transformer-based models and improved the performances of both subtasks to 0.8249 and 0.6775, respectively.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 24
  • May combine two claims. 8
  • The quote only partly states this claim. 2

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The LSTM-CRFs architecture has been widely adopted for named entity recognition tasks in both general and clinical domains.

    “the long short-term memory-conditional random fields (LSTM-CRFs) architecture [25], which is a modified implementation of the recurrent neural network, has been widely adopted for named entity recognition (NER) tasks in both general and clinical domains.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [25] Lample, G, 2016, arXiv source

      The Introduction and Results sections describe an LSTM-CRF architecture for NER and report results in English, Dutch, German, and Spanish. They do not establish that it was widely adopted, and the paper does not report its use in clinical NER; the clinical-domain portion of the claim is unsupported. (LLM, conf=0.87)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    BERT demonstrated promising results in several clinical NLP tasks.

    “and demonstrated promising results in several clinical NLP tasks [27-29].”
    Quote in the paperNot a substantial claim2 of 3 sources checked
    • cites [27] source

      The Results (§4.3; page unavailable) describe BERT as effective on biomedical named entity recognition and report strong performance on a biomedical relation-extraction dataset. The dataset list (§4.1) includes one clinical-text dataset, but the paper does not establish promising BERT results across several specifically clinical NLP tasks; its Conclusion discusses clinical-note and clinical-temporal tasks in connection with BioBERT. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites [28] Alsentzer, E, 2019, arXiv source
    • cites [29] Peng, Y, 2019, arXiv source

  • Classified as probably not a substantial claim.

    Premisecitation

    Previous studies have shown that adopting an ensemble method could further improve clinical NER performances.

    “Previous studies [38-41] have shown that adopting an ensemble method could further improve the clinical NER performances.”
    Quote in the paperSubstantial claim2 of 4 sources checked
    • cites [38] NayelHShashirekhaHImproving NER for Clinical Texts by Ensemble Approach using Segment Representations2017The 14th International Conference on Natural Language Processing (ICON-2017)2017Kolkata, West Bengal, India197204

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [39] source
    • cites [40] source
    • cites [41] source

      The Experimental design section says the authors selected an NER ensemble with the highest F1-score on the development set. However, the Results report improvements for relation extraction and the end-to-end pipeline, not a comparison showing that ensembling improved NER performance over a non-ensemble model. Page numbers are unavailable in our copy. (LLM, conf=0.84)

      Experimental — accuracy not yet measured

  • Classified as probably not a substantial claim.

    Premisecitation

    A previous study reported that generating candidate pairs from all combinations of clinical concepts in document level often generates too many negative samples, causing an extremely imbalanced positive-to-negative sample ratio.

    “However, a previous study [43] has reported that this method often generates too many negative samples (ie, nonrelation), causing an extremely imbalanced positive-to-negative sample ratio.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [43] source

      The Methods (page unavailable) say that considering any two events or times as a candidate pair would generate a huge number of candidate TLinks and might be unsuitable for classifier training. They do not say that those candidates would predominantly be negative samples or establish an extremely imbalanced positive-to-negative ratio. (LLM, conf=0.83)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The 2018 BioCreative/OHNLP Challenge is the first shared task focusing on family history extraction.

    “The 2018 BioCreative/OHNLP Challenge [33,34] is the first shared task focusing on FH extraction.”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites [33] source

      The Introduction and “Task 1: family history extraction” section identify family history extraction as a shared task in the 2018 BioCreative/OHNLP Challenge. Neither section says it was the first shared task focused on family history extraction, so the priority claim is not supported by our copy. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites [34] SijiaLMajidRMYanshanWLiweiWFeichenSSunyangFHongfangLOverview of the BioCreative/OHNLP 2018 Family History Extraction Task20182020-12-10https://www.researchgate.net/publication/327424806_Overview_of_the_BioCreativeOHNLP_2018_Family_History_Extraction_Task

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    During the 2018 BioCreative/OHNLP Challenge, Shi et al explored a joint deep learning approach and achieved the best performance among all participated teams.

    “During that challenge, Shi et al [35] explored a joint deep learning approach and achieved the best performance among all participated teams.”
    Quote in the paperNot a substantial claim1 of 1 sources checked

  • Classified as probably not a substantial claim.

    Premisecitation

    Deep learning-based approaches have demonstrated superior performances in many NLP tasks.

    “More recently, deep learning–based approaches have demonstrated superior performances in many NLP tasks [24].”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [24] Wang, A, 2019, arXiv source

      The Baselines section describes neural models evaluated across nine GLUE language-understanding tasks, and Table 2 reports that some outperform the bag-of-words baseline on several tasks. This supports superior performance in parts of the benchmark, but not the broader claim that deep learning approaches are superior across many NLP tasks generally; the paper also says its best models achieve low absolute scores. (LLM, conf=0.82)

      Experimental — accuracy not yet measured

  • Classified as probably not a substantial claim.

    Premisecitation

    Previous studies developed individual classifiers to handle relations with different cross-distance.

    “Previous studies [44,45] developed individual classifiers to handle relations with different cross-distance”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites [44] source
    • cites [45] source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    A previous study [53] examined various input encoding and output representation of using BERT for relation extraction, and concluded that using representations aggregated from the start position entity markers was the best practice.

    “A previous study [53] examined various input encoding and output representation of using BERT for relation extraction, and concluded that using representations aggregated from the start position entity markers (eg, [S1] and [S2] in Figure 3C) was the best practice.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [53] Soares, L, 2019, arXiv source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    There are preliminary research works demonstrating that generative adversarial networks could be utilized to synthesize clinical text.

    “There are preliminary research works demonstrating that generative adversarial networks could be utilized to synthesize clinical text [56].”
    Quote in the paperNot a substantial claim1 of 1 sources checked

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    The finding that BERT-ner achieved better precision and recall for clinical concept extraction is consistent with a recent study by Si et al [52].

    “We observed that the BERT-ner model achieved both better precision (0.8060 versus 0.7760) and recall (0.8105 versus 0.8087) for clinical concept extraction compared to the LSTM-CRFs, which is consistent with a recent study by Si et al [52].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [52] source

      The Performance Comparison (p. 6) reports higher F1 scores for clinically pretrained BERT models on clinical concept extraction, supporting the broad claim that the citing authors’ result is consistent with Si et al. But that comparison reports F1, not separate precision and recall improvements; the quoted precision and recall values are the citing authors’ own results. (LLM, conf=0.84)

      Experimental — accuracy not yet measured

  • The quote only partly states this claim.

    Findingown data

    After the challenge, the combination of BERT-ner-EN, BERT-cls, and BERT-rel achieved a micro-averaged F1 score of 0.8249 for subtask 1.

    “After the challenge, we further explored the BERT model for NER and the combination of BERT-ner-EN, BERT-cls, and BERT-rel achieved better F1 scores of 0.8249 and 0.6775 for the 2 subtasks, respectively.”
    • Also stated in: § Results · plain
      “After challenge, we further explored new transformer-based models and improved the performances of both subtasks to 0.8249 and 0.6775, respectively.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Our best submission was ranked fifth in subtask 1 of the 2019 n2c2 challenge.

    “Our best submission was ranked fifth in subtask 1 and third in subtask 2.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data

    Our system was among the top-ranked systems (3 out of 21) in the 2019 n2c2 challenge.

    “Our system was among the top-ranked systems (3 out of 21) in the challenge.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In the training set, the cross-distance of candidate pairs ranges from 0 to 10, and 96% of the annotated relations have cross-distances less than 3.

    “In the training set, the cross-distance ranges from 0 to 10 and we found that 96% of the annotated relations have cross-distances less than 3.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Our system identified some observations not annotated in the test set.

    “We also found that our system identified some observations not annotated in the test set.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The "parent" entity only appeared once and the "grandparent" entities appeared 6 times in the training data set.

    “For example, the “parent” entity only appeared once and the “grandparent” entities appeared 6 times in the training data set.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Findingnone given

    After the challenge, we further explored the BERT models for the concept extraction and improved our system in both concept extraction and relation identification.

    “After the challenge, we further explored the BERT models for the concept extraction and improved our system in both concept extraction and relation identification.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    In the sentence "The father also had a history of vascular surgery, a long history of smoking, and has had hip replacement," our system extracted observations of "vascular surgery," "smoking," and "hip replacement," which were annotated in the challenge corpus.

    “For example, in the sentence “The father also had a history of vascular surgery, a long history of smoking, and has had hip replacement,” our system extracted observations of “vascular surgery,” “smoking,” and “hip replacement,” which were annotated in the challenge corpus.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The experimental results showed that the BERT-based classifiers were feasible to determine the family roles, family sides, negation of observations, and living status scores.

    “The experimental results showed that the BERT-based classifiers were feasible to determine the family roles, family sides, negation of observations, and living status scores.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The BERT model is a multilayer transformer encoder model implemented using the self-attention mechanism, pretrained by combining masked language modeling and next sentence prediction.

    “The BERT model is a multilayer transformer encoder model implemented using the self-attention mechanism [48], which is pretrained by combining the masked language modeling method and the next sentence prediction task.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [48] Vaswani, A, 2017, arXiv

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    Extracting family history is more complicated than merely extracting information of the patients because it involves various family members, their living status, and their diseases or disorders, and family members need to be characterized by family role and family side.

    “FH often contains information from different aspects of the patients, including family members, their living status, and their diseases or disorders. Furthermore, patient’s family members need to be characterized by family role (eg, mother) and family side (eg, maternal).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    BERT has two versions: a BASE version with 12 transformer layers and 110 million parameters, and a LARGE version with 24 transformer layers and 340 million parameters.

    “BERT has 2 versions featuring different model sizes, including a BASE version with 12 transformer layers and 110 million parameters, and a LARGE version with 24 transformer layers and 340 million parameters [26].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [26] Devlin, J, 2018, arXiv

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    BERT achieved state-of-the-art performances in 20 NLP benchmarks in the general English domain.

    “a newly emerged bidirectional encoder representations from transformers (BERT) model achieved state-of-the-art performances in 20 NLP benchmarks in the general English domain [26]”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [26] Devlin, J, 2018, arXiv

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Most of the key information for classifying the family member attributes is located in the same sentence where the concepts (ie, family member or observation) are located.

    “One potential reason may be that most of the key information for classifying the family member attributes is located in the same sentence where the concepts (ie, family member or observation) are located.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    Extracting patients' FH information from clinical narratives is a challenging NLP task.

    “Extracting patients’ FH information from clinical narratives is a challenging NLP task.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    Our system is a 2-stage pipeline where the errors generated in the NER will be propagated to relation extraction.

    “Second, our system is a 2-stage pipeline where the errors generated in the NER will be propagated to relation extraction.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationown data

    The representations captured in the special token [CLS] and the representations of the start position markers contain contextual information that is complement to each other.

    “A possible reason is that the representations captured in the special token [CLS] and the representations of the start position markers contain contextual information that is complement to each other.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    The experimental results for the context information strategy were based on fivefold cross-validation on the training set.

    “However, the experimental results based on the fivefold cross-validation on the training set showed that adding the context information did not improve the performance.”
    Quote in the paperSubstantial claim
29 of 29

  • The paper relies on this without support.

    Family history information is not well structured in current electronic health record databases but is often documented as free text in clinical notes.

    “However, FH is not well structured in current electronic health record databases but often documented as free text in clinical notes.”

  • The paper relies on this without support.

    Manually extracting patients' family history information is labor-intensive and time-consuming and cannot be scaled up.

    “Manually extracting patients’ FH information is a labor-intensive and time-consuming procedure that cannot be scaled up.”

  • The paper relies on this without support.

    Machine learning-based methods have better generalizability than rule-based methods for family role and family side classification.

    “here, we applied deep learning–based classification methods as machine learning–based methods have shown a better generalizability.”

  • The paper relies on this without support.

    The BERT model is able to learn both token- and sentence-level representations.

    “as the BERT model is able to learn both token- and sentence-level representations.”

  • The paper relies on this without support.

    Patients’ family history is a critical risk factor associated with numerous diseases.

    “Patients’ FH is a critical risk factor associated with numerous diseases.”

  • The paper relies on this without support.

    Clinical NLP systems that automatically extract family history from clinical narrative are needed for many clinical studies and applications.

    “Clinical NLP systems that automatically extract FH from clinical narrative are needed for many clinical studies and applications.”

  • The paper relies on this without support.

    Family history information has many variations from one patient to another, which makes the development of rules time-consuming and expensive.

    “FH information has many variations from one patient to another, which makes the development of rules time-consuming and expensive.”

  • The paper relies on this without support.

    Machine learning–based models generally have a better generalizability than rule-based systems.

    “Another advantage of our method is that machine learning–based models generally have a better generalizability than rule-based systems and are easy to scale up.”

  • The paper relies on this without support.

    Machine learning–based models are easy to scale up.

    “Another advantage of our method is that machine learning–based models generally have a better generalizability than rule-based systems and are easy to scale up.”
9 of 9

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Examples are described as both unannotated and annotated

    Minor

    The sentence introducing the examples says the observations were not annotated in the test set, while the example says they were annotated in the challenge corpus.

  • Test-set observation counts disagree

    Minor

    Table 1 lists 983 test-set observations, but the gold-standard observation row of the test-set confusion matrix totals 977 (799 + 178). The text does not explain the six-observation difference.

2 of 2

Cites

Flagged first, then by citations

Research works printed but not linked: 17

  1. PortaMA Dictionary of Epidemiology Internet20162020-12-07Oxford, UKOxford University Presshttps://www.oxfordreference.com/view/10.1093/acref/9780199976720.001.0001/acref-9780199976720 · DOI
  2. Meystre, SM, 2008, Yearb Med Inform, 128
  3. Wang, A, 2019, arXiv
  4. Lample, G, 2016, arXiv
  5. Devlin, J, 2018, arXiv
  6. Alsentzer, E, 2019, arXiv
  7. Peng, Y, 2019, arXiv
  8. Wang, Y, 2017, AMIA Jt Summits Transl Sci Proc, 2017, 104
  9. Goryachev, S, 2008, AMIA Annu Symp Proc, 247
  10. Bill, R, 2014, AMIA Annu Symp Proc, 2014, 1709
  11. SijiaLMajidRMYanshanWLiweiWFeichenSSunyangFHongfangLOverview of the BioCreative/OHNLP 2018 Family History Extraction Task20182020-12-10https://www.researchgate.net/publication/327424806_Overview_of_the_BioCreativeOHNLP_2018_Family_History_Extraction_Task
  12. Ramshaw, L, 1995, arXiv
  13. Vaswani, A, 2017, arXiv
  14. Wu, Y, 2018, AMIA Annu Symp Proc, 2018, 1110
  15. Wolf, T, 2020, arXiv
  16. Paszke, A, 2019, arXiv
  17. Soares, L, 2019, arXiv

Cited by

Sorted by citations

Related

Authors
Yang, X., Zhang, H., He, X., Bian, J., & Wu, Y.
Affiliations
University of Florida Health (US)
Corresponding author
Wu, Yonghui
ORCID
Venue
JMIR Medical Informatics
Publisher
JMIR Publications Inc.
Published
2020-12-15
Volume
8
Issue
12
Pages
e22982
ISSN
2291-9694 (ISSN-L)
DOI
10.2196/22982
PMC
PMC7772072
OpenAlex
W3109225475
Semantic Scholar
d929ba38335c7dc382074a14ccab707bf3c79f5a
Cited by
29
References
54
Citations by year
2026: 1 · 2025: 4 · 2024: 6 · 2023: 5 · 2022: 10 · 2021: 3
FWCI
1.24
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • JMIR Medical Informatics (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Europe PMC (PubMed Central) (submittedVersion, cc-by)
  • PubMed Central (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • Patient-Centered Outcomes Research Institute (ME-2018C3-14754)
  • Nvidia
  • University of Florida (UL1 TR001427)
  • National Institutes of Health (U18DP006512, ME-2018C3-14754, R21AG061431, UL1 TR001427)
  • Centers for Disease Control and Prevention (U18DP006512)
  • Clinical and Translational Science Institute, University of Florida (UL1TR001427)
  • National Institute on Aging (R21AG061431, P30AG028740)
  • National Cancer Institute (R01CA246418)
  • National Center for Chronic Disease Prevention and Health Promotion (U18 DP006512)
  • National Center for Advancing Translational Sciences (UL1-TR001427)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
clinical text, clinical information extraction, relation identification, concept extraction, BERT, transformer models, ensemble learning, majority voting, clinical text mining

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-27.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-27.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review