The DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Improvement of intervention information detection for automated clinical literature screening during systematic review
Open accessAbstract
Systematic literature review (SLR) is a crucial method for clinicians and policymakers to make their decisions in a flood of new clinical studies. Because manual literature screening in SLR is a highly laborious task, its automation by natural language processing (NLP) has been welcomed. Although intervention is a key information for literature screening, NLP models for its detection in previous works have not shown adequate performance. In this work, we first design an algorithm for automated construction of high-quality intervention labels by utilizing information retrieved from a clinical trial database. We then design another algorithm for improving model's recall and F1 score by imposing adaptive weights on training instances in the loss function. The intervention detection model trained on the weighted datasets is tested with the Evidence-Based Medicine NLP (EBM-NLP) corpus, and shows 9.7% and 4.0% improvements respectively in recall and F1 score compared to the previous state-of-the-art model on the corpus. The proposed algorithms can boost automation of literature screening during SLR in the clinical domain.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Comparability between models might be further decreased by comparing results between publications that use relaxed vs. strict evaluation approaches for token-based evaluation, or publications that use the same dataset but with different random seeds to split training and testing data.
“Comparability between models might be further decreased by comparing results between publications that use relaxed vs. strict evaluation approaches for token-based evaluation, 34 or publications that use the same dataset but with different random seeds to split training and testing data. 33 , 118”
From Data extraction methods for systematic review (semi)automation: a living systematic review (2021) · cites this paper as [33]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Rethlefsen, M. L., Kirtley, S., Waffenschmidt, S., Ayala, A. P., Moher, D., Page, M. J., Koffel, J. B., PRISMA-S Group, Blunt, H., Brigham, T., et al. (2021) · Systematic Reviews · doi:10.1186/s13643-020-01542-z · Cited by 3,714
Rethlefsen, M. L., Kirtley, S., Waffenschmidt, S., Ayala, A. P., Moher, D., Page, M. J., & Koffel, J. (2019) · doi:10.31219/osf.io/sfc38 · Cited by 3,696
SciBERT: A Pretrained Language Model for Scientific Text
Beltagy, I., Lo, K., & Cohan, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1371 · Cited by 3,218
Descriptive statistics and normality tests for statistical data
Gupta, A., Mishra, P., Pandey, C., Singh, U., Sahu, C., & Keshri, A. (2019) · Annals of Cardiac Anaesthesia · doi:10.4103/aca.aca_157_18 · Cited by 2,628
Egger, M., Jüni, P., Bartlett, C., Holenstein, F., & Sterne, J. (2003) · Health Technology Assessment · doi:10.3310/hta7010 · Cited by 1,181
Borah, R., Brown, A. W., Capers, P. L., & Kaiser, K. A. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-012545 · Cited by 737
Marshall, I. J., Noel‐Storr, A., Kuiper, J., Thomas, J., & Wallace, B. C. (2018) · Research Synthesis Methods · doi:10.1002/jrsm.1287 · Cited by 435
Carroll, C., & Booth, A. (2015) · Research Synthesis Methods · doi:10.1002/jrsm.1128 · Cited by 235
Stylianou, N., Razis, G., Goulis, D. G., & Vlahavas, I. (2020) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2020.101949 · Cited by 22
Flagged first, then by citations
Research works printed but not linked: 21
- Pearson, 2021, How COVID broke the evidence pipeline, Nature, 593, 182, 10.1038/d41586-021-01246-x · DOI
- Elliott, 2021, Decision makers need constantly updated evidence synthesis, Nature, 600, 383, 10.1038/d41586-021-03690-1 · DOI
- Dickersin, 1994, Identifying relevant studies for systematic reviews, BMJ, 309, 1286, 10.1136/bmj.309.6964.1286 · DOI
- Moher, 2009, Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement, PLoS Med., 6, 10.1371/journal.pmed.1000097 · DOI · Paper page
- Polanin, 2019, Best practice guidelines for abstract screening large-evidence systematic reviews and meta-analyses, Res. Synth. Methods., 10, 330, 10.1002/jrsm.1354 · DOI
- McKenzie, 2022, Chapter 3: Defining the criteria for including studies and how they will be grouped for the synthesis
- Marshall, 2019, Toward systematic review automation: a practical guide to using machine learning tools in research synthesis, Syst. Rev., 8, 163, 10.1186/s13643-019-1074-9 · DOI · Paper page
- Kang, 2021, UMLS-based data augmentation for natural language processing of clinical research literature, J. Am. Med. Inform. Assoc., 28, 812, 10.1093/jamia/ocaa309 · DOI
- Schwartz, 2003, A simple algorithm for identifying abbreviation definitions in biomedical text, Pac. Symp. Biocomput., 451
- Sullivan, 2012, Using effect size-or why the p value is not enough, J. Grad. Med. Educ., 4, 279, 10.4300/JGME-D-12-00156.1 · DOI · Paper page
- Hariton, 2018, Randomised controlled trials - the gold standard for effectiveness research: Study design: randomised controlled trials, BJOG, 125, 1716, 10.1111/1471-0528.15199 · DOI · Paper page
- Murad, 2016, New evidence pyramid, Evid. Based. Med., 21, 125, 10.1136/ebmed-2016-110401 · DOI · Paper page
- Paez, 2017, Gray literature: An important resource in systematic reviews, J. Evid. Based Med., 10, 233, 10.1111/jebm.12266 · DOI · Paper page
- Huser, 2012, Precision and negative predictive value of links between ClinicalTrials.gov and PubMed, AMIA Annu. Symp. Proc., 2012, 400
- Pedersen, 2005, High-dose atorvastatin vs usual-dose simvastatin for secondary prevention after myocardial infarction: the IDEAL study: a randomized controlled trial, JAMA, 294, 2437, 10.1001/jama.294.19.2437 · DOI · Paper page
- Doğan, 2014, NCBI disease corpus: a resource for disease name recognition and concept normalization, J. Biomed. Inform., 47, 1, 10.1016/j.jbi.2013.12.006 · DOI
- Andrade, 2015, The primary outcome measure and its importance in clinical trials, J. Clin. Psychiatry, 76, e1320, 10.4088/JCP.15f10377 · DOI
- Bauchner, 2019, Reporting and interpretation of randomized clinical trials, JAMA, 322, 732, 10.1001/jama.2019.12056 · DOI
- Mathieu, 2009, Comparison of registered and published primary outcomes in randomized controlled trials, JAMA, 302, 977, 10.1001/jama.2009.1242 · DOI · Paper page
- Li, 2022, Chapter 5: collecting data
- Wallace, 2016, Extracting PICO sentences from clinical trial reports using supervised distant supervision, J. Mach. Learn. Res., 17, 1
Cited by
Sorted by citations
Data extraction methods for systematic review (semi)automation: a living systematic review
Schmidt, L., Olorisade, B. K., McGuinness, L. A., Thomas, J., & Higgins, J. P. T. (2021) · F1000Research · doi:10.12688/f1000research.51117.1 · Cited by 106
Data extraction methods for systematic review (semi)automation: Update of a living systematic review
Schmidt, L., Finnerty Mutlu, A. N., Elmore, R., Olorisade, B. K., Thomas, J., & Higgins, J. P. T. (2025) · F1000Research · doi:10.12688/f1000research.51117.3 · Cited by 27
Ruan, M., Fan, J., Liu, M., Meng, Z., Zhang, X., & Zhang, C. (2025) · BMC Medical Research Methodology · doi:10.1186/s12874-025-02644-9 · Cited by 12
“The first step aims to eliminate records that are clearly outside the scope of the specific topic, thereby reducing the workload for the full-text screening [14].”
Metadata
- Authors
- Tsubota, T., Bollegala, D., Zhao, Y., Jin, Y., & Kozu, T.
- Affiliations
- Deloitte (United States) (US)
- University of Liverpool (GB)
- Corresponding author
- Tsubota, Tadashi
- Venue
- Journal of Biomedical Informatics
- Publisher
- Elsevier BV
- Published
- 2022-10
- Volume
- 134
- Pages
- 104185
- ISSN
- 1532-0464 (ISSN-L)
- OpenAlex
- W4293216978
- Cited by
- 11
- References
- 11
- Citations by year
- 2026: 1 · 2025: 3 · 2024: 5 · 2023: 2
- FWCI
- 0.26
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of Biomedical Informatics (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- automated literature screening, systematic reviews, natural language processing, clinical trial database, clinical text classification
- MeSH terms
- Algorithms, Automation, Mass Screening, Natural Language Processing, Research Report
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
