The DOI matches its DataCite record; 6 of the 6 references it cites were found in a registry or an index.
Understanding Clinical Trial Reports: Extracting Medical Entities and Their Relations
Open accessAbstract
The best evidence concerning comparative treatment effectiveness comes from clinical trials, the results of which are reported in unstructured articles. Medical experts must manually extract information from articles to inform decision-making, which is time-consuming and expensive. Here we consider the end-to-end task of both (a) extracting treatments and outcomes from full-text articles describing clinical trials (entity identification) and, (b) inferring the reported results for the former with respect to the latter (relation extraction). We introduce new data for this task, and evaluate models that have recently achieved state-of-the-art results on similar tasks in Natural Language Processing. We then propose a new method motivated by how trial results are typically presented that outperforms these purely data-driven baselines. Finally, we run a fielded evaluation of the model with a non-profit seeking to identify existing drugs that might be re-purposed for cancer, showing the potential utility of end-to-end evidence extraction systems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
For relation extraction, F1, precision, and recall remained the most common metrics.
“For relation extraction F1, precision, and recall remained the most common metrics. 62 , 63”
From Data extraction methods for systematic review (semi)automation: a living systematic review (2021) · cites this paper as [63]
A small number of publications did a real-life assessment, where the data extraction algorithm was applied to different, unlabelled, and often much larger datasets, tested while conducting actual systematic reviews, or evaluated in other practical scenarios.
“A small number of publications did a real-life assessment, where the data extraction algorithm was applied to different, unlabelled, and often much larger datasets, tested while conducting actual systematic reviews, or evaluated in other practical scenarios. 46 , 58 , 63 , 69 , 48 , 95 , 101 , 102 , 150 , 178”
From Data extraction methods for systematic review (semi)automation: a living systematic review (2021) · cites this paper as [63]
For relation extraction or normalisation tasks, error-propagation was noted as a practical issue in joint models.
“For relation extraction or normalisation tasks, error-propagation was noted as a practical issue in joint models. 63 , 67”
From Data extraction methods for systematic review (semi)automation: a living systematic review (2021) · cites this paper as [63]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
SciBERT: A Pretrained Language Model for Scientific Text
Beltagy, I., Lo, K., & Cohan, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1371 · Cited by 3,218
Applications of text mining within systematic reviews
Thomas, J., McNaught, J., & Ananiadou, S. (2011) · Research Synthesis Methods · doi:10.1002/jrsm.27 · Cited by 212
PICO Element Detection in Medical Text via Long Short-Term Memory Neural Networks.
Jin, D., & Szolovits, P. (2018) · Proceedings of the BioNLP 2018 workshop · doi:10.18653/v1/w18-2308 · Cited by 61
Evidence inference 2.0: More data, better models.
DeYoung, J., Lehman, E., Nye, B., Marshall, I., & Wallace, B. C. (2020) · Proceedings of the 19th SIGBioMed Workshop on Biomedical Language Processing · doi:10.18653/v1/2020.bionlp-1.13 · Cited by 49
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Data extraction methods for systematic review (semi)automation: a living systematic review
Schmidt, L., Olorisade, B. K., McGuinness, L. A., Thomas, J., & Higgins, J. P. T. (2021) · F1000Research · doi:10.12688/f1000research.51117.1 · Cited by 106
MultiVerS: Improving scientific claim verification with weak supervision and full-document context
Wadden, D., Lo, K., Wang, L., Cohan, A., Beltagy, I., & Hajishirzi, H. (2022) · Findings of the Association for Computational Linguistics: NAACL 2022 · doi:10.18653/v1/2022.findings-naacl.6 · Cited by 60
Data extraction methods for systematic review (semi)automation: Update of a living systematic review
Schmidt, L., Finnerty Mutlu, A. N., Elmore, R., Olorisade, B. K., Thomas, J., & Higgins, J. P. T. (2025) · F1000Research · doi:10.12688/f1000research.51117.3 · Cited by 27
Witte, C., Schmidt, D. M., & Cimiano, P. (2024) · Journal of Biomedical Semantics · doi:10.1186/s13326-024-00305-2 · Cited by 7
A Scoping Review of Adopted Information Extraction Methods for RCTs.
Aletaha, A., Nemati-Anaraki, L., Keshtkar, A., Sedghi, S., Keramatfar, A., & Korolyova, A. (2023) · Medical Journal of The Islamic Republic of Iran · doi:10.47176/mjiri.37.95 · Cited by 2
Metadata
- Authors
- Nye, B. E., DeYoung, J., Lehman, E., Nenkova, A., Marshall, I. J., & Wallace, B. C.
- Affiliations
- Northeastern University (US)
- University of Pennsylvania (US)
- King's College London (GB)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- Volume
- 2021
- Pages
- 485-494
- arXiv
- 2010.03550
- PMC
- PMC8378650
- OpenAlex
- W3092431239
- Cited by
- 15
- References
- 6
- Citations by year
- 2025: 1 · 2024: 1 · 2023: 5 · 2022: 4 · 2021: 4
- FWCI
- 1.05
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Research Portal (King's College London) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1750978)
- National Institutes of Health
- Topics
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Keywords
- clinical trial reports, medical entity extraction, relation extraction, drug repurposing, cancer, evidence extraction
- MeSH terms
- Humans, Natural Language Processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
