CoChat

An annotated corpus of clinical trial publications supporting schema-based relational information extraction.

Open access
Sanchez-Graillet, Olivia, Witte, Christian, Grimm, Frank, Cimiano, Philipp · 2022 · Journal of Biomedical Semantics

The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.

Abstract

Background: The evidence-based medicine paradigm requires the ability to aggregate and compare outcomes of interventions across different trials. This can be facilitated and partially automatized by information extraction systems. In order to support the development of systems that can extract information from published clinical trials at a fine-grained and comprehensive level to populate a knowledge base, we present a richly annotated corpus at two levels. At the first level, entities that describe components of the PICO elements (e.g., population’s age and pre-conditions, dosage of a treatment, etc.) are annotated. The second level comprises schema-level (i.e., slot-filling templates) annotations corresponding to complex PICO elements and other concepts related to a clinical trial (e.g. the relation between an intervention and an arm, the relation between an outcome and an intervention, etc.). Results: The final corpus includes 211 annotated clinical trial abstracts with substantial agreement between annotators at the entity and scheme level. The mean Kappa value for the glaucoma and T2DM corpora was 0.74 and 0.68, respectively, for single entities. The micro-averaged F 1 score to measure inter-annotator agreement for complex entities (i.e. slot-filling templates) was 0.81.The BERT-base baseline method for entity recognition achieved average micro- F 1 scores of 0.76 for glaucoma and 0.77 for diabetes with exact matching. Conclusions: In this work, we have created a corpus that goes beyond the existing clinical trial corpora, since it is annotated in a schematic way that represents the classes and properties defined in an ontology. Although the corpus is small, it has fine-grained annotations and could be used to fine-tune pre-trained machine learning models and transformers to the specific task of extracting information about clinical trial abstracts.For future work, we will use the corpus for training information extraction systems that extract single entities, and predict template slot-fillers (i.e., class data/object properties) to populate a knowledge base that relies on the C-TrO ontology for the description of clinical trials. The resulting corpus and the code to measure inter-annotation agreement and the baseline method are publicly available at https://zenodo.org/record/6365890.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s13326-022-00271-7

Cites

18 of 18 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Annotation scheme based on C-TrO. The annotation schema follows the C-TrO structure with some variants to facilitate the annotation process
Fig. 1 Annotation scheme based on C-TrO. The annotation schema follows the C-TrO structure with some variants to facilitate the annotation processCC BY · PubMed Central
Fig. 2 Example of the annotation of single entities according to the C-TrO schema. (Excerpt taken from abstract PMID 21034177 and annotated with INCEpTION [24] for illustrative purposes)
Fig. 2 Example of the annotation of single entities according to the C-TrO schema. (Excerpt taken from abstract PMID 21034177 and annotated with INCEpTION [24] for illustrative purposes)CC BY · PubMed Central
Fig. 3 Schematic representation of the annotations of an interventional arm. The bimatoprost interventional-arm and its outcome, as reported in the excerpt of abstract PMID 21034177
Fig. 3 Schematic representation of the annotations of an interventional arm. The bimatoprost interventional-arm and its outcome, as reported in the excerpt of abstract PMID 21034177CC BY · PubMed Central
Fig. 4 Classes and properties of the instances of the classes Intervention and Medication. (Intervention_1 and Medication_1, respectively)
Fig. 4 Classes and properties of the instances of the classes Intervention and Medication. (Intervention_1 and Medication_1, respectively)CC BY · PubMed Central
Fig. 5 Templates and slot-fillers corresponding to the instances Intervention_1 and Medication_1. The texts in quotes are annotated text spans
Fig. 5 Templates and slot-fillers corresponding to the instances Intervention_1 and Medication_1. The texts in quotes are annotated text spansCC BY · PubMed Central
Fig. 6 Example of single entities annotated with SANTO (abstract PMID 29110647). The red arrows indicate the annotated entities that are included in the complex entity in Fig. 8
Fig. 6 Example of single entities annotated with SANTO (abstract PMID 29110647). The red arrows indicate the annotated entities that are included in the complex entity in Fig. 8CC BY · PubMed Central
Fig. 7 Example of annotations exported into CoNLL and RDF formats. The CoNLL-style file of entity annotations is at the top and the n-triple file of slot-template annotations is at the bottom
Fig. 7 Example of annotations exported into CoNLL and RDF formats. The CoNLL-style file of entity annotations is at the top and the n-triple file of slot-template annotations is at the bottomCC BY · PubMed Central
Fig. 8 Example of the annotation of the complex entity ClinicalTrial with SANTO (abstract PMID 29110647). The red arrows indicate the single entities (see Fig. 6) that fill the corresponding slots. The slots hasArm, hasPopulation and hasDiffBetweenGroups contain references to other complex entities
Fig. 8 Example of the annotation of the complex entity ClinicalTrial with SANTO (abstract PMID 29110647). The red arrows indicate the single entities (see Fig. 6) that fill the corresponding slots. The slots hasArm, hasPopulation and hasDiffBetweenGroups contain references to other complex entitiesCC BY · PubMed Central
Fig. 9 Example of annotations cases (excerpts from abstracts PubMed 8123096 and 23950156). a overlapping of annotations of the same type, b embedded annotation of the same type (i.e., Drug), and c embedded annotations of different types
Fig. 9 Example of annotations cases (excerpts from abstracts PubMed 8123096 and 23950156). a overlapping of annotations of the same type, b embedded annotation of the same type (i.e., Drug), and c embedded annotations of different typesCC BY · PubMed Central
Fig. 10 Example of the statistics for F1 to determine the IAA on complex entities between two annotators. I: the entity is an Individual. Otherwise, the entity is a literal value
Fig. 10 Example of the statistics for F1 to determine the IAA on complex entities between two annotators. I: the entity is an Individual. Otherwise, the entity is a literal valueCC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 18 references

Research works printed but not linked: 4

  1. Sanchez-Graillet O, Cimiano P, Witte C, Ell B. C-TrO: An Ontology for Summarization and Aggregation of the Level of Evidence in Clinical Trials. In: Proc. of the 5th Joint Ontology Workshops (JOWO): Ontologies and Data in the Life Sciences. Graz: CEUR-WS.org: 2019. http://ceur-ws.org/Vol-2518/paper-ODLS7.pdf.
  2. CoNLL. The SIGNLL Conference on Computational Natural Language Learning. https://www.conll.org/. Accessed 9 Apr 2021.
  3. Hovy E. Annotation. In: Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts: 2010. https://aclanthology.org/P10-5004.
  4. Klie J-C, Bugert M, Boullosa B, de Castilho RE, Gurevych I. The INCEpTION Platform: Machine-Assisted and Knowledge-Oriented Interactive Annotation. In: Proc. of the 27th Int. Conf. on Computational Linguistics: System Demonstrations: 2018. p. 5–9. http://tubiblio.ulb.tu-darmstadt.de/106270/.

Cited by

See all 20

Metadata

Authors
Sanchez-Graillet, O., Witte, C., Grimm, F., & Cimiano, P.
Affiliations
Bielefeld University (DE)
Corresponding author
Sanchez-Graillet, Olivia
Venue
Journal of Biomedical Semantics
Published
2022-05-23
Volume
13
Issue
1
Pages
14-14
ISSN
2041-1480 (ISSN-L)
OpenAlex
W4281293036
Cited by
9
References
17
Citations by year
2025: 4 · 2024: 1 · 2023: 3 · 2022: 1
FWCI
0.73
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of Biomedical Semantics (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • PUB – Publications at Bielefeld University (Bielefeld University) (submittedVersion, cc-by)
Funders
  • Deutsche Forschungsgemeinschaft (376059226 / SPP-1999)
  • Universität Bielefeld
Topics
  • Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
Keywords
clinical trial abstracts, annotated corpus, entity recognition, inter-annotator agreement, glaucoma, type 2 diabetes, BERT-base
MeSH terms
Machine Learning, Clinical Trials as Topic, Glaucoma, Humans, Natural Language Processing, Information Storage and Retrieval, Knowledge Bases

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: An annotated corpus of clinical trial publications supporting schema-based relational information extraction. https://app.cochat.ai/papers/10.1186/s13326-022-00271-7
The paper
Sanchez-Graillet, O., Witte, C., Grimm, F., & Cimiano, P. (2022). An annotated corpus of clinical trial publications supporting schema-based relational information extraction. Journal of Biomedical Semantics. https://doi.org/10.1186/s13326-022-00271-7

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.