An annotated corpus of clinical trial publications supporting schema-based relational information extraction.
Open accessThe DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Abstract
Background: The evidence-based medicine paradigm requires the ability to aggregate and compare outcomes of interventions across different trials. This can be facilitated and partially automatized by information extraction systems. In order to support the development of systems that can extract information from published clinical trials at a fine-grained and comprehensive level to populate a knowledge base, we present a richly annotated corpus at two levels. At the first level, entities that describe components of the PICO elements (e.g., population’s age and pre-conditions, dosage of a treatment, etc.) are annotated. The second level comprises schema-level (i.e., slot-filling templates) annotations corresponding to complex PICO elements and other concepts related to a clinical trial (e.g. the relation between an intervention and an arm, the relation between an outcome and an intervention, etc.). Results: The final corpus includes 211 annotated clinical trial abstracts with substantial agreement between annotators at the entity and scheme level. The mean Kappa value for the glaucoma and T2DM corpora was 0.74 and 0.68, respectively, for single entities. The micro-averaged F 1 score to measure inter-annotator agreement for complex entities (i.e. slot-filling templates) was 0.81.The BERT-base baseline method for entity recognition achieved average micro- F 1 scores of 0.76 for glaucoma and 0.77 for diabetes with exact matching. Conclusions: In this work, we have created a corpus that goes beyond the existing clinical trial corpora, since it is annotated in a schematic way that represents the classes and properties defined in an ontology. Although the corpus is small, it has fine-grained annotations and could be used to fine-tune pre-trained machine learning models and transformers to the specific task of extracting information about clinical trial abstracts.For future work, we will use the corpus for training information extraction systems that extract single entities, and predict template slot-fillers (i.e., class data/object properties) to populate a knowledge base that relies on the C-TrO ontology for the description of clinical trials. The resulting corpus and the code to measure inter-annotation agreement and the baseline method are publicly available at https://zenodo.org/record/6365890.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures

![Fig. 2 Example of the annotation of single entities according to the C-TrO schema. (Excerpt taken from abstract PMID 21034177 and annotated with INCEpTION [24] for illustrative purposes)](https://figures.cochat.ai/paper-figures/14a9ec347585439999be1040a7e2e885/an-annotated-corpus-of-clinical-trial-publications-supporting-schema-based-figure-2.jpg)








Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“400 RCT abstracts in firstcorpus,10k abstract in additional corpus from mined data”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [51]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Micro scores were most widely used, including Al-Hussaini et al. (2022), 32 Sanchez-Graillet et al. (2022), 51 Kim et al. (2011), 52 Verbeke et al. (2012), 53 and Jin and Szolovits (2020) 54”
From Data extraction methods for systematic review (semi)automation: a living systematic review (2021) · cites this paper as [51]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
Assessing Agreement on Classification Tasks: The Kappa Statistic
Carletta, J. (1996) · arXiv · doi:10.48550/arxiv.cmp-lg/9602004 · Cited by 2,088 · cited 2×
“Cohen’s Kappa is calculated as follows: Here, P ( A ) denotes the proportion of times that the two annotators agree, and P ( E ) is the proportion of times that it is expected they agree by chance.”
MethodHopewell, S., Clarke, M., Moher, D., Wager, E., Middleton, P., Altman, D. G., Schulz, K. F., & and the CONSORT Group (2008) · PLoS Medicine · doi:10.1371/journal.pmed.0050020 · Cited by 674 · cited 1×
“The selected abstracts were required to follow the Consolidated Standards of Reporting Trials (CONSORT) [15], which recommend that authors include in the abstracts the compressed structure of the corresponding full article, including background, objective, methodology, results, and conclusions.”
BackgroundResource Description Framework (RDF)
(2009) · Encyclopedia of Library and Information Sciences, Third Edition · doi:10.1081/e-elis3-120043688 · Cited by 316
Answering Clinical Questions with Knowledge-Based and Statistical Techniques
Demner-Fushman, D., & Lin, J. (2007) · Computational Linguistics · doi:10.1162/coli.2007.33.1.63 · Cited by 296
Nye, B., Li, J. J., Patel, R., Yang, Y., Marshall, I., Nenkova, A., & Wallace, B. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1019 · Cited by 224
Combining classifiers for robust PICO element detection.
Boudin, F., Nie, J.-Y., Bartlett, J. C., Grad, R., Pluye, P., & Dawes, M. (2010) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-10-29 · Cited by 131
Research works printed but not linked: 4
- Sanchez-Graillet O, Cimiano P, Witte C, Ell B. C-TrO: An Ontology for Summarization and Aggregation of the Level of Evidence in Clinical Trials. In: Proc. of the 5th Joint Ontology Workshops (JOWO): Ontologies and Data in the Life Sciences. Graz: CEUR-WS.org: 2019. http://ceur-ws.org/Vol-2518/paper-ODLS7.pdf.
- CoNLL. The SIGNLL Conference on Computational Natural Language Learning. https://www.conll.org/. Accessed 9 Apr 2021.
- Hovy E. Annotation. In: Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts: 2010. https://aclanthology.org/P10-5004.
- Klie J-C, Bugert M, Boullosa B, de Castilho RE, Gurevych I. The INCEpTION Platform: Machine-Assisted and Knowledge-Oriented Interactive Annotation. In: Proc. of the 27th Int. Conf. on Computational Linguistics: System Demonstrations: 2018. p. 5–9. http://tubiblio.ulb.tu-darmstadt.de/106270/.
Cited by
Data extraction methods for systematic review (semi)automation: a living systematic review
Schmidt, L., Olorisade, B. K., McGuinness, L. A., Thomas, J., & Higgins, J. P. T. (2021) · F1000Research · doi:10.12688/f1000research.51117.1 · Cited by 106
Data extraction methods for systematic review (semi)automation: Update of a living systematic review
Schmidt, L., Finnerty Mutlu, A. N., Elmore, R., Olorisade, B. K., Thomas, J., & Higgins, J. P. T. (2023) · F1000Research · doi:10.12688/f1000research.51117.2 · Cited by 44
Data extraction methods for systematic review (semi)automation: Update of a living systematic review
Schmidt, L., Finnerty Mutlu, A. N., Elmore, R., Olorisade, B. K., Thomas, J., & Higgins, J. P. T. (2025) · F1000Research · doi:10.12688/f1000research.51117.3 · Cited by 27
The suitability of UMLS and SNOMED-CT for encoding outcome concepts
Newbury, A. M., Liu, H., Idnay, B., & Weng, C. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad161 · Cited by 19
Dhrangadhariya, A., & Müller, H. (2023) · JAMIA Open · doi:10.1093/jamiaopen/ooac107 · Cited by 18
Metadata
- Authors
- Sanchez-Graillet, O., Witte, C., Grimm, F., & Cimiano, P.
- Affiliations
- Bielefeld University (DE)
- Corresponding author
- Sanchez-Graillet, Olivia
- Venue
- Journal of Biomedical Semantics
- Published
- 2022-05-23
- Volume
- 13
- Issue
- 1
- Pages
- 14-14
- ISSN
- 2041-1480 (ISSN-L)
- PMC
- PMC9128209
- OpenAlex
- W4281293036
- Cited by
- 9
- References
- 17
- Citations by year
- 2025: 4 · 2024: 1 · 2023: 3 · 2022: 1
- FWCI
- 0.73
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Journal of Biomedical Semantics (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- PUB – Publications at Bielefeld University (Bielefeld University) (submittedVersion, cc-by)
- Funders
- Deutsche Forschungsgemeinschaft (376059226 / SPP-1999)
- Universität Bielefeld
- Topics
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Keywords
- clinical trial abstracts, annotated corpus, entity recognition, inter-annotator agreement, glaucoma, type 2 diabetes, BERT-base
- MeSH terms
- Machine Learning, Clinical Trials as Topic, Glaucoma, Humans, Natural Language Processing, Information Storage and Retrieval, Knowledge Bases
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: An annotated corpus of clinical trial publications supporting schema-based relational information extraction. https://app.cochat.ai/papers/10.1186/s13326-022-00271-7
- The paper
- Sanchez-Graillet, O., Witte, C., Grimm, F., & Cimiano, P. (2022). An annotated corpus of clinical trial publications supporting schema-based relational information extraction. Journal of Biomedical Semantics. https://doi.org/10.1186/s13326-022-00271-7
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.