Family History Extraction From Synthetic Clinical Narratives Using Natural Language Processing: Overview and Evaluation of a Challenge Data Set and Solutions for the 2019 National NLP Clinical Challenges (n2c2)/Open Health Natural Language Processing (OHNLP) Competition

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

A successful system also needs to consider co-reference resolution, which could be considered a standalone task for NLP systems.

“A successful system also needs to consider co-reference resolution, which could be considered a standalone task for NLP systems [24].”

• cites [24] sourcePartly supported

  The Introduction describes coreference resolution in electronic medical records as a distinct NLP challenge, and the Results evaluate systems on that task. This supports treating it as a standalone task, but the paper does not say that NLP systems generally need coreference resolution to succeed. (LLM, conf=0.82)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

The OHNLP consortium has created multiple deidentified data sets for a couple of NLP tasks based on real clinical sentences.

“Leveraging the research in corpus analysis and deidentification, the Open Health Natural Language Processing (OHNLP) consortium has created multiple deidentified data sets for a couple of NLP tasks based on real clinical sentences [14-16].”

• cites [14] sourceSupported
• cites [15] sourcePartly supported

  The Introduction (p. 3) describes the authors’ creation of MedSTS from clinical sentences, and Data Collection (p. 5) says the sentences were deidentified. The paper describes a resource for one NLP task, semantic textual similarity, with an annotated subset; it does not establish that the OHNLP consortium created multiple datasets for multiple tasks. (LLM, conf=0.86)

  Experimental — accuracy not yet measured
• cites [16] LiuSMojaradMRWangYWangLShenFFuSLiuHOverview of the BioCreative/OHNLP 2018 Family History Extraction TaskProceedings of the BioCreative 2018 Workshop20182021-01-11https://www.researchgate.net/publication/327424806_Overview_of_the_BioCreativeOHNLP_2018_Family_History_Extraction_TaskNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

FH and negation detection are listed as important attributes in clinical information extraction.

“FH and negation detection are listed as important attributes in clinical information extraction [4].”

• cites [4] sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Azab et al developed a data set and a baseline system consisting of narrative answers annotated with family histories from FH questionnaires, based on patient-provided information.

“For example, Azab et al [12] have developed a data set and a baseline system consisting of narrative answers annotated with family histories from FH questionnaires [12], which is based on patient-provided information.”

• cites [12] AzabMDadianSNastaseVAnLMihalceaRTowards extracting medical family history from natural language interactions: A new data set and baselinesProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)20192019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing3–7 November, 2019Hong Kong, ChinaStroudsburg, PAAssociation for Computational Linguistics12551260 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

From previous studies on end-to-end relation extraction tasks, the performance in relation extraction tasks is lower than that in named entity recognition tasks.

“Second, from previous studies on end-to-end relation extraction tasks, the performance in relation extraction tasks is lower than that in named entity recognition tasks [22,23].”

• cites [22] Segura-BedmarIMartínezPHerrero-ZazoMSemEval-2013 Task 9: extraction of drug-drug interactions from biomedical texts (DDIExtraction 2013)2013Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013)June 14-15, 2013Atlanta, GAStroudsburg, PAAssociation for Computational Linguistics341350Not checked yet

  Not checked in the free run.

• cites [23] sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

41 teams from 7 countries signed up for the shared task.

“In summary, 41 teams from 7 countries signed up for this shared task; 17 teams submitted 38 systems for subtask 1 (35 of them were valid) and 9 teams submitted 21 systems (20 of them were valid) for subtask 2.”

May combine two claims.

For subtask 1, son was not detected so well and had the lowest F1 score (0.5926).

“Son was not detected so well, and had the lowest F1 score (0.5926).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

9 teams submitted 21 systems for subtask 2, of which 20 were valid.

“In summary, 41 teams from 7 countries signed up for this shared task; 17 teams submitted 38 systems for subtask 1 (35 of them were valid) and 9 teams submitted 21 systems (20 of them were valid) for subtask 2.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

17 teams submitted 38 systems for subtask 1, of which 35 were valid.

“In summary, 41 teams from 7 countries signed up for this shared task; 17 teams submitted 38 systems for subtask 1 (35 of them were valid) and 9 teams submitted 21 systems (20 of them were valid) for subtask 2.”

Classified as probably not a substantial claim.

The clinical texts in the corpus focus on narrative patient FH information.

“As a result, the clinical texts in the corpus focus on narrative patient FH information.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A total of 99 documents for training and 117 documents for testing were included in the released data set.

“A total of 99 documents for training and 117 documents for testing were included in the released data set.”

May combine two claims.

The IAA between 2 annotators measured before the deidentification process in F1 scores was 0.8324 for subtask 1 and 0.7002 for subtask 2.

“The IAA between 2 annotators measured before the deidentification process in F1 scores was 0.8324 and 0.7002 for subtasks 1 and 2, respectively.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Rule-based approaches were used in some submissions with BERT and NLP techniques for relation extraction.

“In addition, rule-based approaches were used in some submissions with BERT and NLP techniques for relation extraction.”

Probably not a substantial claim: may not pass S3 (load-bearing).

For subtask 2, the ensemble of BERT, deep learning architecture, and some other conventional machine learning algorithms are common strategies adopted by different teams.

“Similar to techniques used for subtask 1, we found that the ensemble of BERT, deep learning architecture, and some other conventional machine learning algorithms are common strategies adopted by different teams.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Some annotators only selected the span to cover the word "suicide" but did not annotate "s", while others did.

“Similarly, taking “suicides” as an example, some annotators only selected the span to cover the word “suicide” but did not annotate “s,” but some other did.”

May combine two claims.

Some systems extracted sister as a correct mention but also extracted paternal aunt as a false-positive case.

“Some systems did extract sister as a correct mention, but paternal aunt was also extracted as a false-positive case.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Some annotators selected the whole phrase as 1 annotation, while others only selected the span for "breast" and "cancer" but overlooked the space in between.

“Taking breast cancer as an example, some annotators selected the whole phrase as 1 annotation, but some others only selected the span for “breast” and “cancer” but overlooked the space in between.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Deep learning architecture with pretrained embeddings was widely used by many teams.

“We also observed that deep learning architecture with pretrained embeddings was widely used by many teams.”

May combine two claims.

In an example, Owen's son was diagnosed with Asperger syndrome and he has no blood relationship with the patient Leo, but some systems extracted Asperger syndrome as the observation incorrectly.

“In the first example above, Owen’s son was diagnosed with Asperger syndrome and he has no blood relationship with the patient Leo. But some systems extracted Asperger syndrome as the observation incorrectly.”

May combine two claims.

The optimal F1 score for subtask 1 and subtask 2 is 0.8745 and 0.6810, respectively.

“The optimal F1 score for subtask 1 and subtask 2 is 0.8745 and 0.6810, respectively.”

Probably not a substantial claim: may not pass S3 (load-bearing).

For subtask 2, fewer submissions were received than for subtask 1.

“For subtask 2, we received fewer submissions and the performance of top 5 systems are shown in Table 9.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A wide variety of methods were used by different teams in both tasks, such as BERT, CNN, Bi-LSTM, CRF, SVM, and rule-based strategies.

“A wide variety of methods were used by different teams in both tasks, such as Bidirectional Encoder Representations from Transformers, convolutional neural network, bidirectional long short-term memory, conditional random field, support vector machine, and rule-based strategies.”

May combine two claims. The quote only partly states this claim.

For observation, common mistakes can be roughly categorized into 2 groups: annotation disagreement or errors made by annotators, and errors made by participants' systems.

“For observation, we roughly categorized the common mistakes into 2 groups. The first group is related to annotation disagreement or errors made by annotators.”

May combine two claims.

Most errors made by participants' systems occurred due to false positives, indicating that those observations/conditions are beyond first or second degree.

“We observed that most of such errors occurred due to false positives, indicating that those observations/conditions are beyond first or second degree.”

Probably not a substantial claim: may not pass S3 (load-bearing).

4 teams incorporated rule-based strategy into their systems for entity identification.

“Besides these, 4 teams incorporated rule-based strategy into their systems for entity identification.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Many teams used state-of-the-art NLP contextual neural language models in their systems, such as BERT and ELMo.

“We found that many teams used the state-of-the-art NLP contextual neural language models in their systems, such as Bidirectional Encoder Representations from Transformers (BERT) [25] and ELMo [26].”

• cites [25] Devlin, J, ArXivNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [26] Peters, ME, ArXivNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Each document in the corpus is annotated by 2 annotators.

“Each document is annotated by 2 annotators, and the whole annotation process is performed by a 5-member annotator team (see the “Acknowledgments” section).”

Probably not a substantial claim: may not pass S3 (load-bearing).

The numeric fields such as dates and phone numbers are manually replaced with synthetic strings.

“The numeric fields such as dates and phone numbers are manually replaced with synthetic strings.”

Probably not a substantial claim: may not pass S3 (load-bearing).

One senior study coordinator worked as the adjudicator to resolve discrepancies between the 2 annotations.

“One senior study coordinator worked as the adjudicator to resolve discrepancies between the 2 annotations.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The observations, family members, and ethnicities are shuffled among the whole corpus to further protect confidentiality.

“To further protect the confidentiality, the observations, family members, and ethnicities are also shuffled among the whole corpus.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Risk assessment of hypertrophic cardiomyopathy considers 1 or more first-degree relatives with a history of sudden cardiac death under age 40 as a significant factor of sudden cardiac death risk in patients with hypertrophic cardiomyopathy.

“For example, the risk assessment of hypertrophic cardiomyopathy considers 1 or more first-degree relatives with a history of sudden cardiac death under age 40 as a significant factor of sudden cardiac death risk in patients with hypertrophic cardiomyopathy [3].”

• cites [3] sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

The Fast Healthcare Interoperability Resources has included FamilyMemberHistory as part of the clinical summary standard.

“The Fast Healthcare Interoperability Resources has also included FamilyMemberHistory as part of the clinical summary standard [13].”

• cites [13] sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

Family history captures shared genetic variations and environmental factors among family members.

“As the key element for precision medicine, family history (FH) captures shared genetic variations and environmental factors among family members [1,2].”

• cites [1] sourceNot checked — no full text of the source
• cites [2] McCarthy, JJ, 2016, Precision Medicine: A Guide to Genomics in Clinical PracticeNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

One of the major sources of FH data is patient-provided information questionnaires, which are usually stored in a semistructured/unstructured format in electronic health records.

“One of the major sources of FH data is patient-provided information questionnaires, which are usually stored in a semistructured/unstructured format in electronic health records [5].”

• cites [5] Wang, Y, 2017, AMIA Jt Summits Transl Sci ProcNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

The patient notes used to curate the corpus were randomly sampled from the Mayo Employee and Community Health cohort.

“The patient notes we used to curate the corpus were randomly sampled from the Mayo Employee and Community Health cohort.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The corpus prepared along with the shared task has encouraged participants internationally to develop FH extraction systems for understanding clinical narratives.

“The corpus we prepared along with the shared task has encouraged participants internationally to develop FH extraction systems for understanding clinical narratives.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The corpus was manually curated from FH sections of clinical notes at Mayo Clinic Rochester and is highly relevant to patients' FH.

“We manually curated the first deidentified clinical narrative from FH sections of clinical notes at Mayo Clinic Rochester, the content of which is highly relevant to patients’ FH.”

Classified as probably not a substantial claim.

Automatically generated semistructured texts were excluded because methods for extracting information from auto-populated formats are significantly different from extracting information from clinical narratives written by human authors.

“Then, we have excluded automatically generated semistructured texts because we expected the methods for extracting information from auto-populated formats to be significantly different from extracting information from clinical narratives written by human authors, with the former requiring more engineering effort than NLP research.”

Classified as probably not a substantial claim.

Sections that combine patients' social history with the FH section were excluded because these have more descriptions of patients' personal social behavior instead of family members.

“We have also excluded sections that combine the patients’ social history with the FH section, as these have more descriptions of patients’ personal social behavior such as occupations and life styles instead of family members.”

Probably not a substantial claim: may not pass S3 (load-bearing).

All the names that appeared in the above examples are synthetic.

“All the names that appeared in the above examples are synthetic.”