A corpus with multi-level annotations of patients, interventions and outcomes to support language processing for medical literature

Nye et al. released the EBM-NLP dataset, comprising approximately 5000 abstracts of RCT reports manually annotated in detail.

“Recently, Nye et al. released the EBM-NLP dataset [23], which comprises ~ 5000 abstracts of RCT reports manually annotated in detail.”

Data extraction systems have been created that can automatically retrieve unstructured information on the trial PICO elements and risk of bias.

“Data extraction systems have been created that can automatically retrieve unstructured information on the trial PICO elements,10–12 and risk of bias.13–16”

The EBM-NLP dataset contains abstracts with manually annotated text snippets describing each element of the PICO.

“This dataset contains abstracts with manually annotated text snippets describing each element of the PICO.”

The PICO extraction model was trained on ~5000 RCT abstracts from the EBM-NLP dataset.

“We trained the model on the ∼5000 RCT abstracts that comprise the EBM-NLP dataset.12”

The PICO span model from Nye et al. achieved F1 scores of 0.71, 0.65, and 0.63 for tagging individual words as describing populations, interventions, and outcomes, respectively.

“This evaluation found F1 scores of 0.71, 0.65, and 0.63 for tagging individual words as describing populations, interventions, and outcomes, respectively.”

Some of the less-frequent data extraction targets in the literature can be categorised as sub-classes of a PICO, for example, by annotating hierarchically multiple entity types such as health condition, age, and gender under the P class.

“Some of the less-frequent data extraction targets in the literature can be categorised as sub-classes of a PICO, 55 for example, by annotating hierarchically multiple entity types such as health condition, age, and gender under the P class.”

Corpora such as the ebm-nlp dataset or PubMed-PICO are available for direct download.

“Corpora such as the ebm-nlp dataset, 55 or PubMed-PICO 54 are available for direct download.”

This problem was solved in more recent publications by conducting a token-based evaluation that computes scores across every single token, hence gaining the ability to score partial matches for multi-word entities.

“This problem was solved in more recent publications by conducting a token-based evaluation that computes scores across every single token, hence gaining the ability to score partial matches for multi-word entities. 55”

Two publications cautioned that cross-validation with a high amount of folds (e.g. 10) causes high variance in evaluation results when using small datasets such as NICTA-PIBOSO.

“Two publications 55 , 85 cautioned that cross-validation with a high amount of folds (e.g. 10) causes high variance in evaluation results when using small datasets such as NICTA-PIBOSO.”

Some publications did not specifically discuss overfitting in the text, but their open-source code indicated that early stopping, dropout, L2-regularisation, or weight decay were used.

“Some publications did not specifically discuss overfitting in the text, but their open-source code indicated that the latter techniques were used. 55 , 75”

Crowd-sourced annotations can contain noisy or incorrect information and have low interrater reliability.

“Crowd-sourced annotations can contain noisy or incorrect information and have low interrater reliability.”

Crowd-sourced annotations can be aggregated to improve quality.

“However, they can be aggregated to improve quality. 55”

In recent publications, partial entity matches (token-wise evaluation) downstream were generally favoured above complete detection, which helps mitigate the impact of noisy annotations on final evaluation scores.

“In recent publications, partial entity matches (i.e., token-wise evaluation) downstream were generally favoured above complete detection, which helps to mitigate this problem’s impact on final evaluation scores. 55 , 83”