The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index; 22 of the 30 sources its claims cite were checked for support.
Trialstreamer: A living, automatically updated database of clinical trial reports
Open accessPublished version of a preprint
Abstract
Objective: Randomized controlled trials (RCTs) are the gold standard method for evaluating whether a treatment works in health care but can be difficult to find and make use of. We describe the development and evaluation of a system to automatically find and categorize all new RCT reports. Materials and Methods: Trialstreamer continuously monitors PubMed and the World Health Organization International Clinical Trials Registry Platform, looking for new RCTs in humans using a validated classifier. We combine machine learning and rule-based methods to extract information from the RCT abstracts, including free-text descriptions of trial PICO (populations, interventions/comparators, and outcomes) elements and map these snippets to normalized MeSH (Medical Subject Headings) vocabulary terms. We additionally identify sample sizes, predict the risk of bias, and extract text conveying key findings. We store all extracted data in a database, which we make freely available for download, and via a search portal, which allows users to enter structured clinical queries. Results are ranked automatically to prioritize larger and higher-quality studies. Results: As of early June 2020, we have indexed 673 191 publications of RCTs, of which 22 363 were published in the first 5 months of 2020 (142 per day). We additionally include 304 111 trial registrations from the International Clinical Trials Registry Platform. The median trial sample size was 66. Conclusions: We present an automated system for finding and categorizing RCTs. This yields a novel resource: a database of structured information automatically extracted for all published RCTs in humans. We make daily updates of this database available on our website (https://trialstreamer.robotreviewer.net).
Overview
CoChat Summary
TL;DR
Trialstreamer is a continuously updated database that automatically identifies and annotates published randomized controlled trials in humans. It combines validated machine learning classifiers with extraction models for PICO elements, sample size, key findings, and risk of bias, and makes the data available via web portal and bulk download.
Key findings
As of June 6, 2020, Trialstreamer indexed 673,191 RCT publications, including 22,363 published in the first 5 months of 2020 (142/day), and included 303,319 ICTRP records (the abstract reports 304,111); median trial sample size was 66 (IQR 30–181). The RCT classifier with a balanced threshold achieved recall 0.94–0.97 and precision ≥0.50, while the human vs nonhuman classifier had near-perfect precision and recall (1.00). Sample size extraction had recall 0.79 and precision 0.88; PICO concept extraction had strict recall 0.73 for populations, 0.78 for interventions, and 0.64 for outcomes, with relaxed recall 0.78, 0.85, and 0.65, respectively. The risk-of-bias model showed good calibration (C-statistic 0.80, Brier score 0.10) but poor binary classification (F1 = 0.45).
Methods
Trialstreamer continuously monitors PubMed and the WHO International Clinical Trials Registry Platform (and medRxiv preprints for COVID-19 during the pandemic), classifies records as RCTs in humans, extracts PICO text spans and MeSH concepts, sample size, key findings, and risk of bias, and stores results in a relational database with a search portal. Models include an ensemble SVM/CNN RCT classifier trained on 280,000 Cochrane Crowd abstracts and evaluated on 49,028 Clinical Hedges abstracts; an SVM human-study classifier trained on 350,364 abstracts and evaluated on 116,789 withheld abstracts; an MLP sample-size extractor trained on 8,935 abstracts and evaluated on 500; an LSTM-CRF PICO span model from EBM-NLP; rule-based PICO concept mapping; and an L2-regularized logistic regression risk-of-bias model trained on 51,429 articles and evaluated on 5,715 withheld articles. Performance was assessed with precision, recall, C-statistic, Brier score, calibration curves, and bootstrap resampling.
Limitations
Stated in the paper: the system recalls 97% of RCTs versus ≈98.5% for systematic review search filters, trading a small fraction of missed articles for higher precision; it monitors only PubMed for publications, so a small proportion of trials in other databases would be missed; risk-of-bias binary classification is poor (recall 0.45), though probabilities are well calibrated for ranking; and bias is not predicted from ICTRP records due to lack of training data. Stated in the paper: PICO concept precision was low in strict evaluation (0.28 for populations, 0.52 for interventions, 0.25 for outcomes), though a post hoc manual re-evaluation found precision of 0.79 for population descriptors and 0.84 for intervention descriptors. Evident from the text: PICO text span metrics are token-level and described as pessimistic, and risk-of-bias predictions use abstracts only rather than full texts.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
14 Valid · 2 Not valid · 5 Not checked · 37 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“On the same validation dataset, we previously found that the manual PubMed Publication Type index had recall of 0.94 and precision of 0.56 for retrieving RCTs.4”
§ Trialstreamer: A living, automatically updated database of clinical trial reports > MATERIALS AND METHODS > Article retrieval
What the sources say
- Supportedcites [4] Marshall, 2018, Machine learning for identifying randomized controlled trials: an evaluation and practitioner’s guide, Res Synth Methods source
The Use cases for RCT identification section reports 93.7% sensitivity (recall) and 56.4% precision for the manually applied PubMed publication-type tag. These round to the claimed 0.94 and 0.56. The paper’s later rerun reports different figures, but the cited figures are present as a previously reported result. (LLM, conf=0.89)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The number of RCTs published accelerates every year,3”
§ Trialstreamer: A living, automatically updated database of clinical trial reports > INTRODUCTION > Background and significance
What the sources say
- Partly supportedcites [3] Bastian, 2010, Seventy-five trials and eleven systematic reviews a day: how will we ever keep up?, PLoS Med sourceExperimental — accuracy not yet measured
The Summary Points and “Where Are We Now?” section report substantial growth in published trials and say the number continues to rise. They do not establish that growth accelerates every year; the latter section notes possible fluctuations in recent annual counts. (LLM, conf=0.86)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“but they still make up a tiny fraction of the contents of health research databases.4”
§ Trialstreamer: A living, automatically updated database of clinical trial reports > INTRODUCTION > Background and significance
What the sources say
- Supportedcites [4] Marshall, 2018, Machine learning for identifying randomized controlled trials: an evaluation and practitioner’s guide, Res Synth Methods source
The Introduction states that RCT articles are a small minority of the available medical literature: 423,000 of 26.6 million PubMed articles (1.6%) were labeled as RCTs in 2016. The Datasets section also says the proportion of RCTs in EMBASE is very low. (LLM, conf=0.95)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Our system recalls 97% of RCTs (as compared with the standard used in systematic review search filters, which achieve ≈98.5% recall).4”
§ Trialstreamer: A living, automatically updated database of clinical trial reports > DISCUSSION
What the sources say
- Supportedcites [4] Marshall, 2018, Machine learning for identifying randomized controlled trials: an evaluation and practitioner’s guide, Res Synth Methods source
Table 2 in Results, “Comparison with traditional filters,” reports 98.5% sensitivity for the Cochrane Highly Sensitive Search Strategy, the high-sensitivity comparator for systematic-review searches. Sensitivity is the recall measure relevant to the claim. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“This evaluation found F1 scores of 0.71, 0.65, and 0.63 for tagging individual words as describing populations, interventions, and outcomes, respectively.”
§ Trialstreamer: A living, automatically updated database of clinical trial reports > RESULTS > Validation of system performance
What the sources say
- Supportedcites Nye et al.12 Nye, 2018, A corpus with multi-level annotations of patients, interventions and outcomes to support language processing for medical literature, Proc Conf Assoc Comput Linguist Meet source
Table 12 (shown in the Conclusions passage) reports test-set F1 scores of 0.71, 0.65, and 0.63 for Participants, Interventions, and Outcomes with the LSTM-CRF span tagger. Section 5.1 explains that the model tags individual tokens as inside or outside PIO spans. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
26 more claims · 2 with their cited sources checked
Create a free account to see all 58 claims and to verify the 3 references still open.
- Probably not a substantial claim. 16
- It may state two separate claims as one. 16
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Around 140 RCTs are published every day, and there are far more potentially problematic papers than assessors can investigate for integrity, highlighting the need to speed up this process [7].”
From Using artificial intelligence to semi-automate trustworthiness assessment of randomized controlled trials: a case study (2025) · cites this paper as [7]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In some instances, however, this evaluation was qualitative where the models were applied to large unlabelled, real-life datasets. 46,58,69,48,95,101,102,164”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [69]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In some instances, however, this evaluation was qualitative where the models were applied to large unlabelled, real-life datasets. 46 , 58 , 69 , 48 , 95 , 101 , 102 , 164”
From Data extraction methods for systematic review (semi)automation: a living systematic review (2021) · cites this paper as [69]
Experimental — accuracy not yet measuredIn the citing paper
“These tools are not intended to replace reference retrieval, but they have been shown to be useful for estimating trends in the literature using minimal human input [19].”
From A narrative review of recent tools and innovations toward automating living systematic reviews and evidence syntheses (2023) · cites this paper as [19]
In the citing paper
“The PubMed PICO Tool [18] and Trialstreamer [19] are both web-based search engines that index the PubMed dataset.”
From A narrative review of recent tools and innovations toward automating living systematic reviews and evidence syntheses (2023) · cites this paper as [19]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Trialstreamer: A living, automatically updated database of clinical trial reports > MATERIALS AND METHODS > Overview
“13 463 abstracts of RCTs with Cochrane Risk of Bias tool assessments in the Cochrane Library: 60% used for training; 40% withheld for evaluation”
§ Trialstreamer: A living, automatically updated database of clinical trial reports > MATERIALS AND METHODS > Automated annotation of RCTs
“We trained this model using the free text from the title and abstract of 51 429 articles and their associated manual “overall” bias assessments as labels; we evaluated the model on 5715 withheld articles.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Trialstreamer: A living, automatically updated database of clinical trial reports > MATERIALS AND METHODS > Overview
“Evaluation on 1497 unseen abstracts with crowd-sourced PICO MeSH labels from Cochrane Crowd”
§ Trialstreamer: A living, automatically updated database of clinical trial reports > MATERIALS AND METHODS > Automated annotation of RCTs
“We evaluate these PICO concept extraction steps against 1437 abstracts that had been manually labelled by the Cochrane Crowd project with structured PICO terms.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Trialstreamer: A living, automatically updated database of clinical trial reports > DISCUSSION
“given recall of 0.45, a substantial portion of high quality studies would be missed in a binary classification mode”
§ Trialstreamer: A living, automatically updated database of clinical trial reports > RESULTS > Validation of system performance
“| Risk of bias (probability of being at low risk of bias) | 0.46 (0.40-0.52) | 0.44 (0.41-0.48) | 0.80 (0.79-0.81) | 0.10 (0.10-0.11) |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Trialstreamer: A living, automatically updated database of clinical trial reports > RESULTS > Validation of system performance
“We found that MeSH labels for PICO concepts had good recall for populations and interventions (recall of 0.78 and 0.85, respectively) but lower precision (0.28 and 0.52, respectively).”
§ Trialstreamer: A living, automatically updated database of clinical trial reports > RESULTS > Validation of system performance
“| Population (n = 1107) | 0.78 (0.76-0.81) | 0.30 (0.29-0.32) | n/a | n/a | | Interventions (n = 954) | 0.85 (0.82, 0.87) | 0.57 (0.54-0.60) | n/a | n/a |”
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817 · cited 1×
“In this prior work, we predicted whether studies’ results might be at risk of bias using the first version of the Cochrane Risk of Bias tool (which examines whether bias is likely due to problems in the random sequence generation, allocation concealment, and blinding, among other issues) (we note that a second version…”
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Dermyshi, E., Wang, Y., Yan, C., Hong, W., Qiu, G., Gong, X., & Zhang, T. (2017) · Neonatology · doi:10.1159/000454668 · Cited by 24,451
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Cochrane Handbook for Systematic Reviews of Interventions
Higgins, J. P. T., Green, S., & Ben Van Den, A. (2020) · International Coaching Psychology Review · doi:10.53841/bpsicpr.2020.15.2.123 · Cited by 5,670
Internal validation of predictive models
Steyerberg, E. W., Harrell, F. E., Borsboom, G. J. J. M., Eijkemans, M. J. C., Vergouwe, Y., & Habbema, J. D. F. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00341-9 · Cited by 2,746
Feature selection, L 1 vs. L 2 regularization, and rotational invariance
Ng, A. Y. (2004) · Twenty-first international conference on Machine learning - ICML '04 · doi:10.1145/1015330.1015435 · Cited by 1,572 · cited 1×
“We used L2 regularization.(26) We trained this model using the free text from the title and abstract of 51 429 articles and their associated manual “overall” bias assessments as labels; we evaluated the model on 5715 withheld articles.”
MethodSeventy-five trials and eleven systematic reviews a day: how will we ever keep up?
Bastian, H., Glasziou, P., & Chalmers, I. (2010) · PLoS Medicine · doi:10.1371/journal.pmed.1000326 · Cited by 1,188
Cited by
Khraisha, Q., Put, S., Kappenberg, J., Warraitch, A., & Hadfield, K. (2023) · arXiv · doi:10.48550/arxiv.2310.17526 · Cited by 268
Khraisha, Q., Put, S., Kappenberg, J., Warraitch, A., & Hadfield, K. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1715 · Cited by 269
“…Some examples of AI tools that have been used in systematic reviews include Rayyan and Abstracker, which help with screening titles and abstracts; 12,13 trialStreamer, which helps with data extraction from full-text articles; 14 and RobotReviewer, which helps with assessing study quality and bias.”
Automated Paper Screening for Clinical Reviews Using Large Language Models
Guo, E., Gupta, M., Deng, J., Park, Y.-J., Paget, M., & Naugler, C. (2023) · arXiv · doi:10.48550/arxiv.2305.00844 · Cited by 230
Automated paper screening for clinical reviews using large language models: Data analysis study
Guo, E., Gupta, M., Deng, J., Park, Y.-J., Paget, M., & Naugler, C. (2024) · Journal of Medical Internet Research · doi:10.2196/48996 · Cited by 205
“Prominent examples of such applications include RobotReviewer [15], TrialStreamer [16], Research Screener [7], DistillerSR [17], and Abstrackr [18], which are artificial intelligence models developed to extract information from scientific articles or abstracts to judge study quality and infer…”
Data extraction methods for systematic review (semi)automation: a living systematic review
Schmidt, L., Olorisade, B. K., McGuinness, L. A., Thomas, J., & Higgins, J. P. T. (2021) · F1000Research · doi:10.12688/f1000research.51117.1 · Cited by 106
Self-citation
Metadata
- Authors
- Marshall, I. J., Nye, B., Kuiper, J., Noel-Storr, A., Marshall, R., Maclean, R., Soboczenski, F., Nenkova, A., Thomas, J., & Wallace, B. C.
- Affiliations
- School of Population Health and Environmental Sciences, King’s College London, London, United Kingdom
- Northeastern University (US)
- Vortext Systems, Groningen, the Netherlands
- University of Oxford (GB)
- Cochrane Editorial and Methods Department, London, United Kingdom
- King's College London (GB)
- University of Pennsylvania (US)
- University College London (GB)
- ORCID
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2020-09-17
- Volume
- 27
- Issue
- 12
- Pages
- 1903-1912
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC7727361
- OpenAlex
- W3087185831
- Cited by
- 83
- References
- 32
- Citations by year
- 2026: 14 · 2025: 19 · 2024: 17 · 2023: 20 · 2022: 8 · 2021: 5
- FWCI
- 3.06
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of the American Medical Informatics Association (publishedVersion, cc-by)
- medRxiv (submittedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- UK Medical Research Council (MR/N015185/1)
- National Institutes of Health under the National Library of Medicine (R01-LM012086)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- Trialstreamer, randomized controlled trials, clinical trial database, automatic information extraction, mesh mapping, clinical trial registration, International Clinical Trials Registry Platform, PubMed, structured clinical data
- MeSH terms
- Data Management, Humans, Bias, Randomized Controlled Trials as Topic, Databases, Factual, Evidence-Based Medicine, Medical Subject Headings, Data Curation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Trialstreamer: A living, automatically updated database of clinical trial reports. https://app.cochat.ai/papers/10.1093/jamia/ocaa163
- The paper
- Marshall, I. J., Nye, B., Kuiper, J., Noel-Storr, A., Marshall, R., Maclean, R., Soboczenski, F., Nenkova, A., Thomas, J., & Wallace, B. C. (2020). Trialstreamer: A living, automatically updated database of clinical trial reports. Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocaa163
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.