Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index; 22 of the 30 sources its claims cite were checked for support.

Trialstreamer: A living, automatically updated database of clinical trial reports

Open access
Marshall, Iain J, Nye, Benjamin, Kuiper, Joël, Noel-Storr, Anna, Marshall, Rachel, Maclean, Rory, Soboczenski, Frank, Nenkova, Ani, Thomas, James, Wallace, Byron C · 2020 · Journal of the American Medical Informatics Association

Published version of a preprint

Abstract

Objective: Randomized controlled trials (RCTs) are the gold standard method for evaluating whether a treatment works in health care but can be difficult to find and make use of. We describe the development and evaluation of a system to automatically find and categorize all new RCT reports. Materials and Methods: Trialstreamer continuously monitors PubMed and the World Health Organization International Clinical Trials Registry Platform, looking for new RCTs in humans using a validated classifier. We combine machine learning and rule-based methods to extract information from the RCT abstracts, including free-text descriptions of trial PICO (populations, interventions/comparators, and outcomes) elements and map these snippets to normalized MeSH (Medical Subject Headings) vocabulary terms. We additionally identify sample sizes, predict the risk of bias, and extract text conveying key findings. We store all extracted data in a database, which we make freely available for download, and via a search portal, which allows users to enter structured clinical queries. Results are ranked automatically to prioritize larger and higher-quality studies. Results: As of early June 2020, we have indexed 673 191 publications of RCTs, of which 22 363 were published in the first 5 months of 2020 (142 per day). We additionally include 304 111 trial registrations from the International Clinical Trials Registry Platform. The median trial sample size was 66. Conclusions: We present an automated system for finding and categorizing RCTs. This yields a novel resource: a database of structured information automatically extracted for all published RCTs in humans. We make daily updates of this database available on our website (https://trialstreamer.robotreviewer.net).

Overview

CoChat Summary

TL;DR

Trialstreamer is a continuously updated database that automatically identifies and annotates published randomized controlled trials in humans. It combines validated machine learning classifiers with extraction models for PICO elements, sample size, key findings, and risk of bias, and makes the data available via web portal and bulk download.

Key findings

As of June 6, 2020, Trialstreamer indexed 673,191 RCT publications, including 22,363 published in the first 5 months of 2020 (142/day), and included 303,319 ICTRP records (the abstract reports 304,111); median trial sample size was 66 (IQR 30–181). The RCT classifier with a balanced threshold achieved recall 0.94–0.97 and precision ≥0.50, while the human vs nonhuman classifier had near-perfect precision and recall (1.00). Sample size extraction had recall 0.79 and precision 0.88; PICO concept extraction had strict recall 0.73 for populations, 0.78 for interventions, and 0.64 for outcomes, with relaxed recall 0.78, 0.85, and 0.65, respectively. The risk-of-bias model showed good calibration (C-statistic 0.80, Brier score 0.10) but poor binary classification (F1 = 0.45).

Methods

Trialstreamer continuously monitors PubMed and the WHO International Clinical Trials Registry Platform (and medRxiv preprints for COVID-19 during the pandemic), classifies records as RCTs in humans, extracts PICO text spans and MeSH concepts, sample size, key findings, and risk of bias, and stores results in a relational database with a search portal. Models include an ensemble SVM/CNN RCT classifier trained on 280,000 Cochrane Crowd abstracts and evaluated on 49,028 Clinical Hedges abstracts; an SVM human-study classifier trained on 350,364 abstracts and evaluated on 116,789 withheld abstracts; an MLP sample-size extractor trained on 8,935 abstracts and evaluated on 500; an LSTM-CRF PICO span model from EBM-NLP; rule-based PICO concept mapping; and an L2-regularized logistic regression risk-of-bias model trained on 51,429 articles and evaluated on 5,715 withheld articles. Performance was assessed with precision, recall, C-statistic, Brier score, calibration curves, and bootstrap resampling.

Limitations

Stated in the paper: the system recalls 97% of RCTs versus ≈98.5% for systematic review search filters, trading a small fraction of missed articles for higher precision; it monitors only PubMed for publications, so a small proportion of trials in other databases would be missed; risk-of-bias binary classification is poor (recall 0.45), though probabilities are well calibrated for ranking; and bias is not predicted from ICTRP records due to lack of training data. Stated in the paper: PICO concept precision was low in strict evaluation (0.28 for populations, 0.52 for interventions, 0.25 for outcomes), though a post hoc manual re-evaluation found precision of 0.79 for population descriptors and 0.84 for intervention descriptors. Evident from the text: PICO text span metrics are token-level and described as pessimistic, and risk-of-bias predictions use abstracts only rather than full texts.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1093/jamia/ocaa163

Cites

33 of 33 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1. How articles are retrieved, annotated, and stored. ICTRP: International Clinical Trials Registry Platform; PICO: populations, interventions/comparators, and outcomes; RCT: randomized controlled trial.
Figure 1. How articles are retrieved, annotated, and stored. ICTRP: International Clinical Trials Registry Platform; PICO: populations, interventions/comparators, and outcomes; RCT: randomized controlled trial.CC BY · PubMed Central
Figure 2. Screenshot of the Trialstreamer web interface home page.
Figure 2. Screenshot of the Trialstreamer web interface home page.CC BY · PubMed Central
Figure 3. Trialstreamer search results.
Figure 3. Trialstreamer search results.CC BY · PubMed Central
Figure 4. Calibration plot of the risk of bias model.
Figure 4. Calibration plot of the risk of bias model.CC BY · PubMed Central
Figure 5. Counts of all randomized controlled trials (RCTs) in PubMed, estimated by manual indexing (yellow) vs automation (blue). CI: confidence interval.
Figure 5. Counts of all randomized controlled trials (RCTs) in PubMed, estimated by manual indexing (yellow) vs automation (blue). CI: confidence interval.CC BY · PubMed Central
Figure 6. Histogram of the number of trial participants in all randomized controlled trials (RCTs) in PubMed, as extracted by our sample size extraction model.
Figure 6. Histogram of the number of trial participants in all randomized controlled trials (RCTs) in PubMed, as extracted by our sample size extraction model.CC BY · PubMed Central

Verification

Claims the paper makes

14 Valid · 2 Not valid · 5 Not checked · 37 Own results and interpretations

26 more claims · 2 with their cited sources checked

Create a free account to see all 58 claims and to verify the 3 references still open.

See all claims — free account
  • Probably not a substantial claim. 16
  • It may state two separate claims as one. 16

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

1 more claim about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

4 of 4

Citations

Cites

Show all 32 references

Cited by

See all 89

Metadata

Authors
Marshall, I. J., Nye, B., Kuiper, J., Noel-Storr, A., Marshall, R., Maclean, R., Soboczenski, F., Nenkova, A., Thomas, J., & Wallace, B. C.
Affiliations
  • School of Population Health and Environmental Sciences, King’s College London, London, United Kingdom
  • Northeastern University (US)
  • Vortext Systems, Groningen, the Netherlands
  • University of Oxford (GB)
  • Cochrane Editorial and Methods Department, London, United Kingdom
  • King's College London (GB)
  • University of Pennsylvania (US)
  • University College London (GB)
Venue
Journal of the American Medical Informatics Association
Published
2020-09-17
Volume
27
Issue
12
Pages
1903-1912
ISSN
  • 1067-5027 (ISSN-L)
  • 1527-974X
OpenAlex
W3087185831
Cited by
83
References
32
Citations by year
2026: 14 · 2025: 19 · 2024: 17 · 2023: 20 · 2022: 8 · 2021: 5
FWCI
3.06
Type
journal article
Language
en
Open access
Locations
  • Journal of the American Medical Informatics Association (publishedVersion, cc-by)
  • medRxiv (submittedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Funders
  • UK Medical Research Council (MR/N015185/1)
  • National Institutes of Health under the National Library of Medicine (R01-LM012086)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
Keywords
Trialstreamer, randomized controlled trials, clinical trial database, automatic information extraction, mesh mapping, clinical trial registration, International Clinical Trials Registry Platform, PubMed, structured clinical data
MeSH terms
Data Management, Humans, Bias, Randomized Controlled Trials as Topic, Databases, Factual, Evidence-Based Medicine, Medical Subject Headings, Data Curation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Trialstreamer: A living, automatically updated database of clinical trial reports. https://app.cochat.ai/papers/10.1093/jamia/ocaa163
The paper
Marshall, I. J., Nye, B., Kuiper, J., Noel-Storr, A., Marshall, R., Maclean, R., Soboczenski, F., Nenkova, A., Thomas, J., & Wallace, B. C. (2020). Trialstreamer: A living, automatically updated database of clinical trial reports. Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocaa163

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.