The DOI matches its Crossref record; 68 of the 68 references it cites were found in a registry or an index.
Comparing generative and extractive approaches to information extraction from abstracts describing randomized clinical trials.
Open accessAbstract
Background: Systematic reviews of Randomized Controlled Trials (RCTs) are an important part of the evidence-based medicine paradigm. However, the creation of such systematic reviews by clinical experts is costly as well as time-consuming, and results can get quickly outdated after publication. Most RCTs are structured based on the Patient, Intervention, Comparison, Outcomes (PICO) framework and there exist many approaches which aim to extract PICO elements automatically. The automatic extraction of PICO information from RCTs has the potential to significantly speed up the creation process of systematic reviews and this way also benefit the field of evidence-based medicine. Results: Previous work has addressed the extraction of PICO elements as the task of identifying relevant text spans or sentences, but without populating a structured representation of a trial. In contrast, in this work, we treat PICO elements as structured templates with slots to do justice to the complex nature of the information they represent. We present two different approaches to extract this structured information from the abstracts of RCTs. The first approach is an extractive approach based on our previous work that is extended to capture full document representations as well as by a clustering step to infer the number of instances of each template type. The second approach is a generative approach based on a seq2seq model that encodes the abstract describing the RCT and uses a decoder to infer a structured representation of a trial including its arms, treatments, endpoints and outcomes. Both approaches are evaluated with different base models on a manually annotated dataset consisting of RCT abstracts on an existing dataset comprising 211 annotated clinical trial abstracts for Type 2 Diabetes and Glaucoma. For both diseases, the extractive approach (with ) reached the best $$F_1$$ F 1 score, i.e. 0.547 ( $$\pm 0.006$$ ± 0.006 ) for type 2 diabetes and 0.636 ( $$\pm 0.006$$ ± 0.006 ) for glaucoma. Generally, the $$F_1$$ F 1 scores were higher for glaucoma than for type 2 diabetes and the standard deviation was higher for the generative approach. Conclusion: In our experiments, both approaches show promising performance extracting structured PICO information from RCTs, especially considering that most related work focuses on the far easier task of predicting less structured objects. In our experimental results, the extractive approach performs best in both cases, although the lead is greater for glaucoma than for type 2 diabetes. For future work, it remains to be investigated how the base model size affects the performance of both approaches in comparison. Although the extractive approach currently leaves more room for direct improvements, the generative approach might benefit from larger models.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Witte 2024 176 who report that their F1-score is higher on glaucoma studies compared with diabetes 2 studies (0.63 vs. 0.54)”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [176]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Evidence based medicine: what it is and what it isn't
Sackett, D. L., Rosenberg, W. M. C., Gray, J. A. M., Haynes, R. B., & Richardson, W. S. (1996) · BMJ · doi:10.1136/bmj.312.7023.71 · Cited by 15,769
Larian Studios Dev Team (2024) · doi:10.1145/3292500.3330701 · Cited by 8,618
Deep Learning using Rectified Linear Units (ReLU)
Agarap, A. F. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.08375 · Cited by 2,399
Longformer: The Long-Document Transformer
Beltagy, I., Peters, M. E., & Cohan, A. (2020) · arXiv · doi:10.48550/arxiv.2004.05150 · Cited by 2,281
Text Chunking Using Transformation-Based Learning
Ramshaw, L. A., & Marcus, M. P. (1999) · Text, Speech and Language Technology · doi:10.1007/978-94-017-2390-9_10 · Cited by 1,460
Scaling Instruction-Finetuned Language Models
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.11416 · Cited by 1,441 · cited 5×
“…• We present an extension of the approach proposed by Witte and Cimiano [14] in two directions: i) relying on Longformers [16] and Flan-T5 [17] to encode the complete abstract and infer templates and slots for the complete document jointly, and ii) using a clustering step to cluster the…”
MethodInfluentialSeventy-five trials and eleven systematic reviews a day: how will we ever keep up?
Bastian, H., Glasziou, P., & Chalmers, I. (2010) · PLoS Medicine · doi:10.1371/journal.pmed.1000326 · Cited by 1,188
How Quickly Do Systematic Reviews Go Out of Date? A Survival Analysis
Shojania, K. G., Sampson, M., Ansari, M. T., Ji, J., Doucette, S., & Moher, D. (2007) · Annals of Internal Medicine · doi:10.7326/0003-4819-147-4-200708210-00179 · Cited by 838
Research works printed but not linked: 2
- Sanchez-Graillet O, Cimiano P, Witte C, Ell B. C-TrO: An Ontology for Summarization and Aggregation of the Level of Evidence in Clinical Trials. In: Proc. of the 5th Joint Ontology Workshops (JOWO): Ontologies and Data in the Life Sciences. 2019. https://ceur-ws.org/Vol-2518/paper-ODLS7.pdf.
- Jin D, Szolovits P. PICO Element Detection in Medical Text via Long Short-Term Memory Neural Networks. In: Proceedings of the BioNLP 2018 Workshop. Melbourne: Association for Computational Linguistics; 2018. p. 67–75. https://doi.org/10.18653/v1/W18-2308. · DOI · Paper page
Cited by
Data extraction methods for systematic review (semi)automation: a living systematic review
Schmidt, L., Olorisade, B. K., McGuinness, L. A., Thomas, J., & Higgins, J. P. T. (2021) · F1000Research · doi:10.12688/f1000research.51117.1 · Cited by 106
Data extraction methods for systematic review (semi)automation: Update of a living systematic review
Schmidt, L., Finnerty Mutlu, A. N., Elmore, R., Olorisade, B. K., Thomas, J., & Higgins, J. P. T. (2025) · F1000Research · doi:10.12688/f1000research.51117.3 · Cited by 27
Schmidt, D. M., & Cimiano, P. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1406857 · Cited by 15
Self-citationArtificial intelligence for clinical trial design, conduct, and analysis: a narrative review
Knapen, D. G., van Kruchten, M., de Groot, D. J. A., Broekman, K. E., & Fehrmann, R. S. N. (2026) · ESMO Real World Data and Digital Oncology · doi:10.1016/j.esmorw.2026.100682 · Cited by 13
Natural Language Processing in Support of Evidence-based Medicine: A Scoping Review
Xu, Z., Ma, H., Ding, Y., Zhang, G., Weng, C., & Peng, Y. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.1103 · Cited by 11
Metadata
- Authors
- Witte, C., Schmidt, D. M., & Cimiano, P.
- Affiliations
- Bielefeld University (DE)
- Corresponding author
- Schmidt, David M.
- Venue
- Journal of Biomedical Semantics
- Published
- 2024-04-23
- Volume
- 15
- Issue
- 1
- Pages
- 3-3
- ISSN
- 2041-1480 (ISSN-L)
- PMC
- PMC11036632
- OpenAlex
- W4395039840
- Cited by
- 7
- References
- 62
- Citations by year
- 2026: 3 · 2025: 3 · 2024: 1
- FWCI
- 0.95
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Journal of Biomedical Semantics (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- PUB – Publications at Bielefeld University (Bielefeld University) (submittedVersion, cc-by)
- Funders
- Federal Ministry of Health (BMG), Germany (KINBIOTICS)
- Ministry of Culture and Science of the State of North Rhine-Westphalia, Germany (NW21-059A (SAIL))
- Deutsche Forschungsgemeinschaft (Transregional Collaborative Research Centre TRR 318 "Constructing Explainability" (Projects B1 and C5))
- Universität Bielefeld
- Topics
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Keywords
- randomized controlled trials, clinical trial abstracts, structured data extraction, sequence-to-sequence models, template-based extraction, systematic reviews, type 2 diabetes, glaucoma, clinical text mining, document clustering, PICO framework, evidence-based medicine, clinical trial comparison
- MeSH terms
- Abstracting and Indexing, Humans, Natural Language Processing, Randomized Controlled Trials as Topic, Information Storage and Retrieval
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Comparing generative and extractive approaches to information extraction from abstracts describing randomized clinical trials. https://app.cochat.ai/papers/10.1186/s13326-024-00305-2
- The paper
- Witte, C., Schmidt, D. M., & Cimiano, P. (2024). Comparing generative and extractive approaches to information extraction from abstracts describing randomized clinical trials. Journal of Biomedical Semantics. https://doi.org/10.1186/s13326-024-00305-2
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.