Trialstreamer: A living, automatically updated database of clinical trial reports
Open accessPublished version of a preprint
The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index; 22 of the 30 sources its claims cite were checked for support.
Abstract
Objective: Randomized controlled trials (RCTs) are the gold standard method for evaluating whether a treatment works in health care but can be difficult to find and make use of. We describe the development and evaluation of a system to automatically find and categorize all new RCT reports. Materials and Methods: Trialstreamer continuously monitors PubMed and the World Health Organization International Clinical Trials Registry Platform, looking for new RCTs in humans using a validated classifier. We combine machine learning and rule-based methods to extract information from the RCT abstracts, including free-text descriptions of trial PICO (populations, interventions/comparators, and outcomes) elements and map these snippets to normalized MeSH (Medical Subject Headings) vocabulary terms. We additionally identify sample sizes, predict the risk of bias, and extract text conveying key findings. We store all extracted data in a database, which we make freely available for download, and via a search portal, which allows users to enter structured clinical queries. Results are ranked automatically to prioritize larger and higher-quality studies. Results: As of early June 2020, we have indexed 673 191 publications of RCTs, of which 22 363 were published in the first 5 months of 2020 (142 per day). We additionally include 304 111 trial registrations from the International Clinical Trials Registry Platform. The median trial sample size was 66. Conclusions: We present an automated system for finding and categorizing RCTs. This yields a novel resource: a database of structured information automatically extracted for all published RCTs in humans. We make daily updates of this database available on our website (https://trialstreamer.robotreviewer.net).
Overview
CoChat Summary
TL;DR
Trialstreamer is a continuously updated database that automatically identifies and annotates published randomized controlled trials in humans. It combines validated machine learning classifiers with extraction models for PICO elements, sample size, key findings, and risk of bias, and makes the data available via web portal and bulk download.
Key findings
As of June 6, 2020, Trialstreamer indexed 673,191 RCT publications, including 22,363 published in the first 5 months of 2020 (142/day), and included 303,319 ICTRP records (the abstract reports 304,111); median trial sample size was 66 (IQR 30–181). The RCT classifier with a balanced threshold achieved recall 0.94–0.97 and precision ≥0.50, while the human vs nonhuman classifier had near-perfect precision and recall (1.00). Sample size extraction had recall 0.79 and precision 0.88; PICO concept extraction had strict recall 0.73 for populations, 0.78 for interventions, and 0.64 for outcomes, with relaxed recall 0.78, 0.85, and 0.65, respectively. The risk-of-bias model showed good calibration (C-statistic 0.80, Brier score 0.10) but poor binary classification (F1 = 0.45).
Methods
Trialstreamer continuously monitors PubMed and the WHO International Clinical Trials Registry Platform (and medRxiv preprints for COVID-19 during the pandemic), classifies records as RCTs in humans, extracts PICO text spans and MeSH concepts, sample size, key findings, and risk of bias, and stores results in a relational database with a search portal. Models include an ensemble SVM/CNN RCT classifier trained on 280,000 Cochrane Crowd abstracts and evaluated on 49,028 Clinical Hedges abstracts; an SVM human-study classifier trained on 350,364 abstracts and evaluated on 116,789 withheld abstracts; an MLP sample-size extractor trained on 8,935 abstracts and evaluated on 500; an LSTM-CRF PICO span model from EBM-NLP; rule-based PICO concept mapping; and an L2-regularized logistic regression risk-of-bias model trained on 51,429 articles and evaluated on 5,715 withheld articles. Performance was assessed with precision, recall, C-statistic, Brier score, calibration curves, and bootstrap resampling.
Limitations
Stated in the paper: the system recalls 97% of RCTs versus ≈98.5% for systematic review search filters, trading a small fraction of missed articles for higher precision; it monitors only PubMed for publications, so a small proportion of trials in other databases would be missed; risk-of-bias binary classification is poor (recall 0.45), though probabilities are well calibrated for ranking; and bias is not predicted from ICTRP records due to lack of training data. Stated in the paper: PICO concept precision was low in strict evaluation (0.28 for populations, 0.52 for interventions, 0.25 for outcomes), though a post hoc manual re-evaluation found precision of 0.79 for population descriptors and 0.84 for intervention descriptors. Evident from the text: PICO text span metrics are token-level and described as pessimistic, and risk-of-bias predictions use abstracts only rather than full texts.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
14 Valid · 2 Not valid · 5 Not checked · 37 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“On the same validation dataset, we previously found that the manual PubMed Publication Type index had recall of 0.94 and precision of 0.56 for retrieving RCTs.4”
§ Trialstreamer: A living, automatically updated database of clinical trial reports > MATERIALS AND METHODS > Article retrieval
What the sources say
- Supportedcites [4] Marshall, 2018, Machine learning for identifying randomized controlled trials: an evaluation and practitioner’s guide, Res Synth Methods source
The Use cases for RCT identification section reports 93.7% sensitivity (recall) and 56.4% precision for the manually applied PubMed publication-type tag. These round to the claimed 0.94 and 0.56. The paper’s later rerun reports different figures, but the cited figures are present as a previously reported result. (LLM, conf=0.89)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The number of RCTs published accelerates every year,3”
§ Trialstreamer: A living, automatically updated database of clinical trial reports > INTRODUCTION > Background and significance
What the sources say
- Partly supportedcites [3] Bastian, 2010, Seventy-five trials and eleven systematic reviews a day: how will we ever keep up?, PLoS Med sourceExperimental — accuracy not yet measured
The Summary Points and “Where Are We Now?” section report substantial growth in published trials and say the number continues to rise. They do not establish that growth accelerates every year; the latter section notes possible fluctuations in recent annual counts. (LLM, conf=0.86)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“but they still make up a tiny fraction of the contents of health research databases.4”
§ Trialstreamer: A living, automatically updated database of clinical trial reports > INTRODUCTION > Background and significance
What the sources say
- Supportedcites [4] Marshall, 2018, Machine learning for identifying randomized controlled trials: an evaluation and practitioner’s guide, Res Synth Methods source
The Introduction states that RCT articles are a small minority of the available medical literature: 423,000 of 26.6 million PubMed articles (1.6%) were labeled as RCTs in 2016. The Datasets section also says the proportion of RCTs in EMBASE is very low. (LLM, conf=0.95)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Our system recalls 97% of RCTs (as compared with the standard used in systematic review search filters, which achieve ≈98.5% recall).4”
§ Trialstreamer: A living, automatically updated database of clinical trial reports > DISCUSSION
What the sources say
- Supportedcites [4] Marshall, 2018, Machine learning for identifying randomized controlled trials: an evaluation and practitioner’s guide, Res Synth Methods source
Table 2 in Results, “Comparison with traditional filters,” reports 98.5% sensitivity for the Cochrane Highly Sensitive Search Strategy, the high-sensitivity comparator for systematic-review searches. Sensitivity is the recall measure relevant to the claim. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“This evaluation found F1 scores of 0.71, 0.65, and 0.63 for tagging individual words as describing populations, interventions, and outcomes, respectively.”
§ Trialstreamer: A living, automatically updated database of clinical trial reports > RESULTS > Validation of system performance
What the sources say
- Supportedcites Nye et al.12 Nye, 2018, A corpus with multi-level annotations of patients, interventions and outcomes to support language processing for medical literature, Proc Conf Assoc Comput Linguist Meet source
Table 12 (shown in the Conclusions passage) reports test-set F1 scores of 0.71, 0.65, and 0.63 for Participants, Interventions, and Outcomes with the LSTM-CRF span tagger. Section 5.1 explains that the model tags individual tokens as inside or outside PIO spans. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
26 more claims · 2 with their cited sources checked
Create a free account to see all 58 claims and to verify the 3 references still open.
- Probably not a substantial claim. 16
- It may state two separate claims as one. 16
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Around 140 RCTs are published every day, and there are far more potentially problematic papers than assessors can investigate for integrity, highlighting the need to speed up this process [7].”
From Using artificial intelligence to semi-automate trustworthiness assessment of randomized controlled trials: a case study (2025) · cites this paper as [7]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In some instances, however, this evaluation was qualitative where the models were applied to large unlabelled, real-life datasets. 46,58,69,48,95,101,102,164”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [69]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In some instances, however, this evaluation was qualitative where the models were applied to large unlabelled, real-life datasets. 46 , 58 , 69 , 48 , 95 , 101 , 102 , 164”
From Data extraction methods for systematic review (semi)automation: a living systematic review (2021) · cites this paper as [69]
Experimental — accuracy not yet measuredIn the citing paper
“These tools are not intended to replace reference retrieval, but they have been shown to be useful for estimating trends in the literature using minimal human input [19].”
From A narrative review of recent tools and innovations toward automating living systematic reviews and evidence syntheses (2023) · cites this paper as [19]
In the citing paper
“The PubMed PICO Tool [18] and Trialstreamer [19] are both web-based search engines that index the PubMed dataset.”
From A narrative review of recent tools and innovations toward automating living systematic reviews and evidence syntheses (2023) · cites this paper as [19]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Trialstreamer: A living, automatically updated database of clinical trial reports > MATERIALS AND METHODS > Overview
“13 463 abstracts of RCTs with Cochrane Risk of Bias tool assessments in the Cochrane Library: 60% used for training; 40% withheld for evaluation”
§ Trialstreamer: A living, automatically updated database of clinical trial reports > MATERIALS AND METHODS > Automated annotation of RCTs
“We trained this model using the free text from the title and abstract of 51 429 articles and their associated manual “overall” bias assessments as labels; we evaluated the model on 5715 withheld articles.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Trialstreamer: A living, automatically updated database of clinical trial reports > MATERIALS AND METHODS > Overview
“Evaluation on 1497 unseen abstracts with crowd-sourced PICO MeSH labels from Cochrane Crowd”
§ Trialstreamer: A living, automatically updated database of clinical trial reports > MATERIALS AND METHODS > Automated annotation of RCTs
“We evaluate these PICO concept extraction steps against 1437 abstracts that had been manually labelled by the Cochrane Crowd project with structured PICO terms.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Trialstreamer: A living, automatically updated database of clinical trial reports > DISCUSSION
“given recall of 0.45, a substantial portion of high quality studies would be missed in a binary classification mode”
§ Trialstreamer: A living, automatically updated database of clinical trial reports > RESULTS > Validation of system performance
“| Risk of bias (probability of being at low risk of bias) | 0.46 (0.40-0.52) | 0.44 (0.41-0.48) | 0.80 (0.79-0.81) | 0.10 (0.10-0.11) |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Trialstreamer: A living, automatically updated database of clinical trial reports > RESULTS > Validation of system performance
“We found that MeSH labels for PICO concepts had good recall for populations and interventions (recall of 0.78 and 0.85, respectively) but lower precision (0.28 and 0.52, respectively).”
§ Trialstreamer: A living, automatically updated database of clinical trial reports > RESULTS > Validation of system performance
“| Population (n = 1107) | 0.78 (0.76-0.81) | 0.30 (0.29-0.32) | n/a | n/a | | Interventions (n = 954) | 0.85 (0.82, 0.87) | 0.57 (0.54-0.60) | n/a | n/a |”
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817 · cited 1×
“In this prior work, we predicted whether studies’ results might be at risk of bias using the first version of the Cochrane Risk of Bias tool (which examines whether bias is likely due to problems in the random sequence generation, allocation concealment, and blinding, among other issues) (we note that a second version…”
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Dermyshi, E., Wang, Y., Yan, C., Hong, W., Qiu, G., Gong, X., & Zhang, T. (2017) · Neonatology · doi:10.1159/000454668 · Cited by 24,451
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Cochrane Handbook for Systematic Reviews of Interventions
Higgins, J. P. T., Green, S., & Ben Van Den, A. (2020) · International Coaching Psychology Review · doi:10.53841/bpsicpr.2020.15.2.123 · Cited by 5,670
Internal validation of predictive models
Steyerberg, E. W., Harrell, F. E., Borsboom, G. J. J. M., Eijkemans, M. J. C., Vergouwe, Y., & Habbema, J. D. F. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00341-9 · Cited by 2,746
Feature selection, L 1 vs. L 2 regularization, and rotational invariance
Ng, A. Y. (2004) · Twenty-first international conference on Machine learning - ICML '04 · doi:10.1145/1015330.1015435 · Cited by 1,572 · cited 1×
“We used L2 regularization.(26) We trained this model using the free text from the title and abstract of 51 429 articles and their associated manual “overall” bias assessments as labels; we evaluated the model on 5715 withheld articles.”
MethodSeventy-five trials and eleven systematic reviews a day: how will we ever keep up?
Bastian, H., Glasziou, P., & Chalmers, I. (2010) · PLoS Medicine · doi:10.1371/journal.pmed.1000326 · Cited by 1,188
Cited by
Khraisha, Q., Put, S., Kappenberg, J., Warraitch, A., & Hadfield, K. (2023) · arXiv · doi:10.48550/arxiv.2310.17526 · Cited by 268
Khraisha, Q., Put, S., Kappenberg, J., Warraitch, A., & Hadfield, K. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1715 · Cited by 269
“…Some examples of AI tools that have been used in systematic reviews include Rayyan and Abstracker, which help with screening titles and abstracts; 12,13 trialStreamer, which helps with data extraction from full-text articles; 14 and RobotReviewer, which helps with assessing study quality and bias.”
Automated Paper Screening for Clinical Reviews Using Large Language Models
Guo, E., Gupta, M., Deng, J., Park, Y.-J., Paget, M., & Naugler, C. (2023) · arXiv · doi:10.48550/arxiv.2305.00844 · Cited by 230
Automated paper screening for clinical reviews using large language models: Data analysis study
Guo, E., Gupta, M., Deng, J., Park, Y.-J., Paget, M., & Naugler, C. (2024) · Journal of Medical Internet Research · doi:10.2196/48996 · Cited by 205
“Prominent examples of such applications include RobotReviewer [15], TrialStreamer [16], Research Screener [7], DistillerSR [17], and Abstrackr [18], which are artificial intelligence models developed to extract information from scientific articles or abstracts to judge study quality and infer…”
Data extraction methods for systematic review (semi)automation: a living systematic review
Schmidt, L., Olorisade, B. K., McGuinness, L. A., Thomas, J., & Higgins, J. P. T. (2021) · F1000Research · doi:10.12688/f1000research.51117.1 · Cited by 106
Self-citationOfori-Boateng, R., Aceves-Martins, M., Wiratunga, N., & Moreno-Garcia, C. F. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10844-w · Cited by 86
Santos, Á. O. D., da Silva, E. S., Couto, L. M., Reis, G. V. L., & Belo, V. S. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104389 · Cited by 64
Data extraction methods for systematic review (semi)automation: Update of a living systematic review
Schmidt, L., Finnerty Mutlu, A. N., Elmore, R., Olorisade, B. K., Thomas, J., & Higgins, J. P. T. (2023) · F1000Research · doi:10.12688/f1000research.51117.2 · Cited by 44
Self-citationAdvancing Chinese biomedical text mining with community challenges
Zong, H., Wu, R., Cha, J., Feng, W., Wu, E., Li, J., Shao, A., Tao, L., Li, Z., Tang, B., et al. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104716 · Cited by 44
“For example, mining the gene-disease semantic associations from literatures can help researchers in discovering novel gene candidates related to diseases [93], while extracting PICOS information from the literature can contribute to the construction of clinical evidence knowledge bases [94].”
BackgroundInterpreting Randomized Controlled Trials
Msaouel, P., Lee, J., & Thall, P. F. (2023) · Cancers · doi:10.3390/cancers15194674 · Cited by 39
“Worldwide, results of approximately 140 RCTs are published each day, comprising an immense compendium of data that can be daunting for clinical practitioners and other stakeholders to digest efficiently [5].”
BackgroundSchmidt, L., Sinyor, M., Webb, R. T., Marshall, C., Knipe, D., Eyles, E. C., John, A., Gunnell, D., & Higgins, J. P. T. (2023) · Zeitschrift für Evidenz, Fortbildung und Qualität im Gesundheitswesen · doi:10.1016/j.zefq.2023.06.007 · Cited by 31
Living Health Technology Assessment: Issues, Challenges and Opportunities
Thokala, P., Srivastava, T., Smith, R. A., Ren, S., Whittington, M. D., Elvidge, J., Wong, R., & Uttley, L. (2023) · PharmacoEconomics · doi:10.1007/s40273-022-01229-4 · Cited by 44
“Trialstreamer [27] combines both trial identification (using RobotSearch [24]) and data extraction (using RobotReviewer [26]) to maintain a new and publicly available living database of all healthcare RCTs in humans, from PubMed and the WHO International Clinical Trials Registry Platform.”
MethodPICO entity extraction for preclinical animal literature.
Wang, Q., Liao, J., Lapata, M., & Macleod, M. (2022) · Systematic Reviews · doi:10.1186/s13643-022-02074-4 · Cited by 29
“For example, Trialstreamer [35] could be used to identify interventions and outcomes in preclinical experiments.”
MethodArtificial intelligence in food and nutrition evidence: The challenges and opportunities
Bailey, R. L., MacFarlane, A. J., Field, M. S., Tagkopoulos, I., Baranzini, S. E., Edwards, K. M., Rose, C. J., Schork, N. J., Singhal, A., Wallace, B., et al. (2024) · PNAS Nexus · doi:10.1093/pnasnexus/pgae461 · Cited by 36
“An available prototype, Trialstreamer, is a publicly available living repository of all articles describing RCTs in humans that makes RCT data fully computable (Figure 4) (64, 67).”
Self-citationŠuster, S., Baldwin, T., & Verspoor, K. M. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1749 · Cited by 40
Data extraction methods for systematic review (semi)automation: Update of a living systematic review
Schmidt, L., Finnerty Mutlu, A. N., Elmore, R., Olorisade, B. K., Thomas, J., & Higgins, J. P. T. (2025) · F1000Research · doi:10.12688/f1000research.51117.3 · Cited by 27
“One, for example, applied their system to new, unlabelled data and reported that classifying the whole of PubMed takes around 20 hours using a graphics processing unit (GPU).(39) In another example, the authors reported using Google Colab GPUs, along with estimates of computing time for different training settings.”
InfluentialSelf-citationState of the evidence: a survey of global disparities in clinical trials
Marshall, I. J., L’Esperance, V., Marshall, I. J., Thomas, J., Noel-Storr, A. H., Soboczenski, F., Nye, B., Nenkova, A., & Wallace, B. (2021) · BMJ Global Health · doi:10.1136/bmjgh-2020-004145 · Cited by 38
Self-citationA question-answering framework for automated abstract screening using large language models
Akinseloyin, O., Jiang, X., & Palade, V. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae166 · Cited by 28
“26 Examples like RobotReviewer 27,28 and TrialStreamer 29 showcased the power of integrating AI into the review process, with Robot-Reviewer claiming to reach accuracy comparable to human reviewers.”
BackgroundSobczyk, M., Zheng, J., Smith, G. D., & Gaunt, T. R. (2023) · BMJ Open · doi:10.1136/bmjopen-2023-072087 · Cited by 28
“The conclusions of MR and RCT studies did not align well, with MR analyses providing evidence for reduced risk of MS conferred by higher vitamin D levels, but no significant therapeutic effect of vitamin D in existing MS was found in the five small meta-analysed trials.”
ResultMarshall, I. J., Trikalinos, T. A., Soboczenski, F., Yun, H. S., Kell, G., Marshall, R., & Wallace, B. C. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2022.08.013 · Cited by 23
“We have described the NLP data extraction methods used in Trialstreamer in detail previously.[17].”
MethodInfluentialSelf-citationDigital Tools to Support the Systematic Review Process: An Introduction
Schmidt, L., Cree, I. A., Campbell, F., & WCT EVI MAP group (2025) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.70100 · Cited by 27
Generative Artificial Intelligence in Medicine
Shanmugam, D., Agrawal, M., Movva, R., Chen, I. Y., Ghassemi, M., Jacobs, M. L., & Pierson, E. (2025) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-103123-095332 · Cited by 41
Methodology reporting improved over time in 176,469 randomized controlled trials
Kilicoglu, H., Jiang, L., Hoang, L., Mayo-Wilson, E., Vinkers, C. H., & Otte, W. M. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.08.004 · Cited by 19
Windisch, P. Y., Dennstädt, F., Koechli, C., Schröder, C., Aebersold, D. M., Forster, R. E., & Zwahlen, D. R. (2024) · Cureus · doi:10.7759/cureus.75748 · Cited by 24
“…information, e.g., according to PICO (patient, intervention, control, outcome) characteristics, could improve various processes such as screening the literature for relevant publications, assessing adherence to reporting guidelines, and ultimately automating the process of evidence synthesis [4,5].”
Srinivasan, A., Berkowitz, J. S., Friedrich, N. A., Kivelson, S., & Tatonetti, N. P. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2025.29418 · Cited by 37
(2024) · doi:10.1101/2024.03.31.24305138 · Cited by 18
“Other research has focused on automating risk of bias . assessment using text classification 35–37 and rhetorical classification of medical abstracts (e.g., Objective, Methods) 38-40 .”
MethodJiang, L., Lan, M., Menke, J. D., Vorland, C. J., & Kilicoglu, H. (2024) · Scientific Reports · doi:10.1038/s41598-024-72130-7 · Cited by 18
“Most error cases involved sentences predicted to report a CONSORT item different from the original label (37.2% of errors).”
BackgroundMethodInfluentialSaady, M., Eissa, M., Yacoub, A. S., Hamed, A. B., & Azzazy, H. M. E.-S. (2025) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2025.103066 · Cited by 15
Šuster, S., Baldwin, T., & Verspoor, K. M. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.04.006 · Cited by 16
A neuro-symbolic method for understanding free-text medical evidence.
Kang, T., Turfah, A., Kim, J., Perotte, A., & Weng, C. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocab077 · Cited by 11
Christopoulou, S. C. (2022) · BioMedInformatics · doi:10.3390/biomedinformatics2030032 · Cited by 11
ContReviews: A content-based recommendation system for updating Living Evidences in health care
Tenti, P., Thomas, J., Peñaloza, R., & Pasi, G. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.112981 · Cited by 11
Self-citationAutomatically Summarizing Evidence from Clinical Trials: A Prototype Highlighting Current Challenges
Ramprasad, S., Mcinerney, J., Marshall, I. J., & Wallace, B. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-demo.27 · Cited by 18
Self-citationRedHOT: A Corpus of Annotated Medical Questions, Experiences, and Claims on Social Media
Wadhwa, S., Khetan, V., Amir, S., & Wallace, B. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.61 · Cited by 25
Self-citationDennstädt, F., Windisch, P. Y., Filchenko, I., et al. (2024) · medRxiv · doi:10.1101/2024.12.03.24318390 · Cited by 8
Evaluating the Efficacy of Large Language Models for Systematic Review and Meta-Analysis ScreeningPreprint
Luo, R., Sastimoglu, Z., Faisal, A. I., & Deen, M. J. (2024) · medRxiv · doi:10.1101/2024.06.03.24308405 · Cited by 17
“These tools, alongside innovations like RobotReviewer [47], TrialStreamer [48], and .”
Methodological information extraction from randomized controlled trial publications: a pilot study.
Hoang, L., Guan, Y., & Kilicoglu, H. (2022) · PubMed Central · Cited by 10
Schmidt, L., Campbell, F., Craig, D., Walker, V. R., Rooney, A. A., Schmitt, C., Härtung, T., & Thomas, J. (2025) · Evidence-Based Toxicology · doi:10.1080/2833373x.2025.2546791 · Cited by 13
Self-citationShi, X., Zhao, W., Chen, T., Yang, C., & Du, J. (2025) · Nature Communications · doi:10.1038/s41467-025-62783-x · Cited by 17
“However, these methods are predominantly based on the less speci fi c framework of evidence-based medicine, which emphasizes Population-Intervention-Comparator-Outcome (PICO) related concepts, such as Trialstreamer and the EvidenceMap 18,19 .”
Jiang, L., Vorland, C. J., Ying, X., Brown, A. W., Menke, J. D., Hong, G., Lan, M., Mayo‐Wilson, E., & Kilicoglu, H. (2025) · Scientific Data · doi:10.1038/s41597-025-04629-1 · Cited by 11
“…sentences in abstracts or full-text articles by PICO elements (Population, Intervention, Comparator, and Outcome) for article screening 18–21 , extracting PICO-related or other methodological terms to aid data extraction 22–28 , and classifying text for automated risk-of-bias assessment 29–31 .”
Windisch, P. Y., Dennstädt, F., Koechli, C., Foerster, R., Schröder, C., Aebersold, D. M., & Zwahlen, D. R. (2024) · JCO Clinical Cancer Informatics · doi:10.1200/cci-24-00150 · Cited by 5
Guo, E., Gupta, M., Deng, J., et al. (2023) · doi:10.2196/preprints.48996 · Cited by 7
Barnett, A. (2022) · F1000Research · doi:10.12688/f1000research.123002.1 · Cited by 7
“23 I downloaded a list of published randomised trials from the trialstreamer web page 24 using the PubMed Central ID (PMCID).”
Constructing a finer-grained representation of clinical trial results from ClinicalTrials.gov
Shi, X., & Du, J. (2024) · Scientific Data · doi:10.1038/s41597-023-02869-7 · Cited by 9
“TrialStreamer 12 and TrialSummarizer 13 are tools that extracts results from clinical trials in a large scale.”
RoBIn: A Transformer-based model for risk of bias inference with machine reading comprehension
Dias, A. C., Moreira, V. P., & Comba, J. L. D. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104819 · Cited by 12
Au, L. S., Qu, L., Nielsen, J., Ge, Z., Gurrin, L. C., Mol, B. W., & Wang, R. (2025) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2025.111672 · Cited by 9
Reconsidering what makes syntheses of psychological intervention studies useful
Sakaluk, J. K., Santis, C. D., Kilshaw, R. E., Pittelkow, M.-M., Brandes, C. M., Boness, C. L., Botanov, Y., Williams, A. J., Wendt, D. C., Lorenzo‐Luaces, L., et al. (2023) · Nature Reviews Psychology · doi:10.1038/s44159-023-00213-9 · Cited by 5
Dennstädt, F., Windisch, P. Y., Filchenko, I., Zink, J., Putora, P. M., Shaheen, A., Gaio, R., Čihorić, N., Wosny, M., Aeppli, S., et al. (2025) · Oncology · doi:10.1159/000546946 · Cited by 16
High-precision information retrieval for rapid clinical guideline updates
Borchert, F., Wullenweber, P., Oeser, A., Kreuzberger, N., Karge, T., Langer, T. G., Skoetz, N., Wieler, L. H., Schapranow, M.-P., & Arnrich, B. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01648-5 · Cited by 11
“However, for different surveillance strategies, ongoing clinical trials or results posted on Clin-icalTrials.gov prior to a journal publication might be equally relevant.”
Windisch, P. Y., Dennstädt, F., Koechli, C., et al. (2024) · medRxiv · doi:10.1101/2024.07.01.24309767 · Cited by 4
A Novel Question-Answering Framework for Automated Abstract Screening Using Large Language ModelsPreprint
Akinseloyin, O., Jiang, X., & Palade, V. (2023) · medRxiv · doi:10.1101/2023.12.17.23300102 · Cited by 4
State of the evidence: a survey of global disparities in clinical trialsPreprint
Marshall, I. J., L’Esperence, V., Marshall, I. J., et al. (2020) · medRxiv · doi:10.1101/2020.10.08.20209353 · Cited by 4
“HIV/AIDS and sexually transmitted infections 8 (7–9) [15] 7 (6–8) [15] 3 (3–4) [17] 2,500 [13] 41 [11] 130 [21] 2,300 [8]”
MethodInfluentialSelf-citationWindisch, P. Y., Dennstädt, F., Koechli, C., Schroeder, C., Aebersold, D. M., Foerster, R., & Zwahlen, D. R. (2024) · medRxiv · doi:10.1101/2024.10.23.24316005 · Cited by 3
Modernizing evidence synthesis for evidence-based medicine
Saldanha, I. J., Adam, G. P., Schmid, C. H., Trikalinos, T. A., & Konnyu, K. J. (2023) · Elsevier eBooks · doi:10.1016/b978-0-323-91200-6.00006-1 · Cited by 4
Wang, W., Xu, B., Fang, T., Zhang, L., & Song, Y. (2023) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2023.semeval-1.1 · Cited by 9
“This makes NLP systems more practical and feasible to support the large-scale interpretation and retrieval of medical evidence (Marshall et al., 2020; Molinet et al., 2022; Pradeep et al., 2022; Yasunaga et al., 2022).”
BackgroundSobczyk, M. K., Zheng, J., Smith, G. D., & Gaunt, T. R. (2022) · medRxiv · doi:10.1101/2022.04.11.22273633 · Cited by 3
Controversial Trials First: Identifying Disagreement Between Clinical Guidelines and New Evidence.
Borchert, F., Meister, L., Langer, T. G., Follmann, M., Arnrich, B., & Schapranow, M.-P. (2021) · PubMed · Cited by 5
Language Models and Food–Health Evidence: Challenges, Opportunities, and Implications
Jackson, D., Gousiopoulos, A., & Soldatos, T. G. (2026) · BioMedInformatics · doi:10.3390/biomedinformatics6020013 · Cited by 3
“…on such structured resources, and combining modern AI capabilities with ‘traditional’ bioinformatics TM approaches (e.g., like in [12,49,153–159]) could accelerate the development of more effective biomedical applications for personalized nutrition or food knowledge synthesis (e.g.,…”
Advances in orofacial pain research: a bibliometric analysis
Qin, S., Liu, J., Fan, J., Qin, Z., & Jia, J. (2025) · Frontiers in Neurology · doi:10.3389/fneur.2025.1609437 · Cited by 3
Predicting the sample size of randomized controlled trials using natural language processing
Windisch, P. Y., Dennstädt, F., Koechli, C., Forster, R. E., Schröder, C., Aebersold, D. M., & Zwahlen, D. R. (2024) · JAMIA Open · doi:10.1093/jamiaopen/ooae116 · Cited by 5
Windisch, P., Dennstädt, F., Koechli, C., Förster, R., Schröder, C., Aebersold, D. M., & Zwahlen, D. R. (2024) · doi:10.1101/2024.07.09.24310155 · Cited by 2
“5,9 This makes sense for them, as their approach is to extract the sample size if it is explicitly mentioned in the paper.”
BackgroundA Scoping Review of Adopted Information Extraction Methods for RCTs.
Aletaha, A., Nemati-Anaraki, L., Keshtkar, A., Sedghi, S., Keramatfar, A., & Korolyova, A. (2023) · Medical Journal of The Islamic Republic of Iran · doi:10.47176/mjiri.37.95 · Cited by 2
Interpreting Randomized Controlled TrialsPreprint
Msaouel, P., Lee, J., & Thall, P. F. (2023) · Preprints.org · doi:10.20944/preprints202309.0093.v1 · Cited by 2
Gendrin, A., Souliotis, L., Loudon-Griffiths, J., Aggarwal, R., Amoako, D., Desouza, G., Dimitrievska, S., Metcalfe, P., Louvet, E., & Sahni, H. (2023) · JMIR Formative Research · doi:10.2196/44876 · Cited by 5
“NLP tools have been widely applied to extract information from electronic health records [5-15,16,17,53-55].”
BackgroundMethodAutomated detection of over- and under-dispersion in baseline tables in randomised controlled trialsPreprint
Barnett, A. G. (2022) · Research Square · doi:10.21203/rs.3.rs-1496802/v1 · Cited by 2
Starck, T., Ravaud, P., & Boutron, I. (2025) · medRxiv · doi:10.64898/2025.12.19.25342674 · Cited by 1
The Impact of Artificial Intelligence on Radiology: Opportunities, Challenges, and Future Directions
Hamilton, C. E. (2024) · International Journal of Innovative Science and Research Technology (IJISRT) · doi:10.38124/ijisrt/ijisrt24aug1512 · Cited by 1
Windisch, P. Y., Dennstädt, F., Koechli, C., Forster, R. E., Schröder, C., Aebersold, D. M., & Zwahlen, D. R. (2024) · medRxiv · doi:10.1101/2024.06.17.24309020 · Cited by 1
“4 When it comes to inclusion and exclusion criteria, they are often able to identify and extract the paragraphs that provide the information.”
BackgroundShi, X., Zhao, W., Chen, T., Yang, C., & Du, J. (2024) · medRxiv · doi:10.1101/2024.03.18.24304457 · Cited by 1
The importance of appraising articles and conducting psychiatric journal clubs
Miller, E., Weightman, M. J., Basu, A., & Amos, A. J. (2023) · Australasian Psychiatry · doi:10.1177/10398562231191678 · Cited by 1
Promoting Fairness in Classification of Quality of Medical Evidence
Šuster, S., Baldwin, T., & Verspoor, K. M. (2023) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2023.bionlp-1.39 · Cited by 2
“For assessing overall RoB in a single clinical study, we broadly follow the approach in Marshall et al. (2020).”
BackgroundMethodInfluentialGrames, C., Zakarian, P., Franquemont, C., Cabrera, A., & Elsissy, J. G. (2026) · medRxiv · doi:10.64898/2026.09.28.26364191
Multimodal Survival Analysis on Hierarchical Representation of Clinical Trial Data
Das, J., Mamann, A., Bugiotti, F., et al. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-39823-9_11
Panigrahi, S. K., & Bhattacharya, S. (2026) · The Indian Journal of Medical Research · doi:10.25259/ijmr_3469_2026
One pyramid of hierarchy of study designs
Doke, P. (2026) · The Indian Journal of Medical Research · doi:10.25259/ijmr_813_2026
Костюк, І. А. (2026) · Farmatsevtychnyi zhurnal · doi:10.32352/0367-3057.4.26.05
Zorcic, K., Bartsch, E., Lim, B., McCallum, G., Bakel, T. V., Hacker, A., & Fralick, M. (2026) · PLoS ONE · doi:10.1371/journal.pone.0351135
Li, X., Yin, C., Mpofu, E., & Liu, X. (2026) · Geriatric Nursing · doi:10.1016/j.gerinurse.2026.104195
Wang, D., Datta, S., Glasgow, J., Lee, K., Paek, H., Zhang, J., Zheng, Y., Huang, Y.-L., He, L., Rastegar-Mojarad, M., et al. (2026) · JMIR AI · doi:10.2196/81049 · Cited by 1
“Several existing SLR automation tools have been developed [38], including Trialstreamer [39], SWIFT-Review [40], DistillerSR [41], NestedKnowledge [42], SWIFT-ActiveScreener [43], Abstrackr [44], EPPI-Reviewer [45], and RobotReviewer [46].”
Evidence-Bound Generation for Medicines Information with Verifiable Claims and Visible Uncertainty
Ferraro, L., Ricci, M., (1560388), G. M., & Greco, P. (2026) · Archives of Pharmacy Practice · doi:10.51847/tzcg1ksqsz
Windisch, P. Y., Dennstädt, F., Koechli, C., Foerster, R., Schröder, C., Aebersold, D. M., & Zwahlen, D. R. (2025) · Oncology · doi:10.1159/000546970 · Cited by 5
Automating Risk of Bias Inference in Clinical Studies
Dias, A. C., Moreira, V. P., & Comba, J. L. D. (2025) · doi:10.5753/sbcas_estendido.2025.7404
Jiang, L., Vorland, C. J., Ying, X., Brown, A. W., Menke, J. D., Hong, G., Lan, M., Mayo‐Wilson, E., & Halil Kilicoglu, B. (2025) · medRxiv · doi:10.1101/2025.01.14.25320543
Desde la necesidad de información a la estrategia de búsqueda, pasando por el formato PICO
Lima, A. O. D. L., Osa, S. T.-L., & Martín-Ruiz, E. (2024) · Ars Pharmaceutica (Internet) · doi:10.30827/ars.v66i1.31474 · Cited by 2
Next Generation Evidence: High-Precision Information Retrieval for Rapid Clinical Guideline UpdatesPreprint
Borchert, F., Wullenweber, P., Oeser, A., Kreuzberger, N., Karge, T., Langer, T. G., Skoetz, N., Wieler, L. H., Schapranow, M.-P., & Bert Arnrich, A. (2024) · medRxiv · doi:10.1101/2024.12.02.24318184
“…controlled trials (RCTs) in PubMed , from which we automatically extract information related to PICO elements, inspired by the Trialstreamer [23] system, • All registered clinical trials in ClinicalTrials.gov , which are accessed through the AACT database [24], as well as • Curated…”
InfluentialChoudhari, R. (2024) · Innovative Practice in Breast Health · doi:10.1016/j.ibreh.2024.100013 · Cited by 1
Constructing a Finer-Grained Representation of Clinical Trial Results from ClinicalTrials.govPreprint
Shi, X., & Du, J. (2023) · medRxiv · doi:10.1101/2023.10.25.23297572
What does the evidence say? Automatic information extraction from medical trial reports
Nye, B. (2022) · Digital Repository Service (Northeastern University) · doi:10.17760/d20467219
Self-citationA PICO-based Knowledge Graph for Representing Clinical Evidence.
Bai, Y., Sun, H., & Du, J. (2021) · Cited by 1
Metadata
- Authors
- Marshall, I. J., Nye, B., Kuiper, J., Noel-Storr, A., Marshall, R., Maclean, R., Soboczenski, F., Nenkova, A., Thomas, J., & Wallace, B. C.
- Affiliations
- School of Population Health and Environmental Sciences, King’s College London, London, United Kingdom
- Northeastern University (US)
- Vortext Systems, Groningen, the Netherlands
- University of Oxford (GB)
- Cochrane Editorial and Methods Department, London, United Kingdom
- King's College London (GB)
- University of Pennsylvania (US)
- University College London (GB)
- ORCID
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2020-09-17
- Volume
- 27
- Issue
- 12
- Pages
- 1903-1912
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC7727361
- OpenAlex
- W3087185831
- Cited by
- 83
- References
- 32
- Citations by year
- 2026: 14 · 2025: 19 · 2024: 17 · 2023: 20 · 2022: 8 · 2021: 5
- FWCI
- 3.06
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of the American Medical Informatics Association (publishedVersion, cc-by)
- medRxiv (submittedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- UK Medical Research Council (MR/N015185/1)
- National Institutes of Health under the National Library of Medicine (R01-LM012086)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- Trialstreamer, randomized controlled trials, clinical trial database, automatic information extraction, mesh mapping, clinical trial registration, International Clinical Trials Registry Platform, PubMed, structured clinical data
- MeSH terms
- Data Management, Humans, Bias, Randomized Controlled Trials as Topic, Databases, Factual, Evidence-Based Medicine, Medical Subject Headings, Data Curation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.