Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 35 of the 36 references it cites were found in a registry or an index.

Machine learning to assist risk-of-bias assessments in systematic reviews

Open access
Millard, Louise AC, Flach, Peter A, Higgins, Julian PT · 2016 · International Journal of Epidemiology

Abstract

BACKGROUND: Risk-of-bias assessments are now a standard component of systematic reviews. At present, reviewers need to manually identify relevant parts of research articles for a set of methodological elements that affect the risk of bias, in order to make a risk-of-bias judgement for each of these elements. We investigate the use of text mining methods to automate risk-of-bias assessments in systematic reviews. We aim to identify relevant sentences within the text of included articles, to rank articles by risk of bias and to reduce the number of risk-of-bias assessments that the reviewers need to perform by hand. METHODS: We use supervised machine learning to train two types of models, for each of the three risk-of-bias properties of sequence generation, allocation concealment and blinding. The first model predicts whether a sentence in a research article contains relevant information. The second model predicts a risk-of-bias value for each research article. We use logistic regression, where each independent variable is the frequency of a word in a sentence or article, respectively. RESULTS: We found that sentences can be successfully ranked by relevance with area under the receiver operating characteristic (ROC) curve (AUC) > 0.98. Articles can be ranked by risk of bias with AUC > 0.72. We estimate that more than 33% of articles can be assessed by just one reviewer, where two reviewers are normally required. CONCLUSIONS: We show that text mining can be used to assist risk-of-bias assessments.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/ije/dyv306

Cites

35 of 35 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1. Venn diagram of data in our dataset, showing the number of articles with a value for each property.
Figure 1. Venn diagram of data in our dataset, showing the number of articles with a value for each property.CC BY · PubMed Central
Figure 2. Data from the Cochrane risk of bias tool. Dotted line represents a relationship that we need to infer.
Figure 2. Data from the Cochrane risk of bias tool. Dotted line represents a relationship that we need to infer.CC BY · PubMed Central
Figure 3. Flow diagram illustrating dataset creation process, from Cochrane risk-of-bias data to labelled dataset.
Figure 3. Flow diagram illustrating dataset creation process, from Cochrane risk-of-bias data to labelled dataset.CC BY · PubMed Central
Figure 4. Flow diagram illustrating process from PDF article to predicted scores. The article model predicts a score denoting the risk of bias, and the sentence model predicts a score denoting the relevance of a sentence, for each risk of bias property.
Figure 4. Flow diagram illustrating process from PDF article to predicted scores. The article model predicts a score denoting the risk of bias, and the sentence model predicts a score denoting the relevance of a sentence, for each risk of bias property.CC BY · PubMed Central
Figure 5. ROC curves for sentence-level learning comparing results using the relevant/not dataset, compared with the train relevant/rest, test relevant/not results, for each risk-of-bias property. Test B: blue dashed curve. Test C: green solid curve. Generated with parametric rate-oriented method 33 with associated point-wise confidence bounds. This method requires a constant number of examples of each label (low,not-low) in each fold, so we add examples to make N constant, and use random selection of examples to correct these frequencies, for instance by removing a randomly selected positive example and duplicating a randomly selected negative example in a particular fold. TPR: true positive rate; FPR: false positive rate.
Figure 5. ROC curves for sentence-level learning comparing results using the relevant/not dataset, compared with the train relevant/rest, test relevant/not results, for each risk-of-bias property. Test B: blue dashed curve. Test C: green solid curve. Generated with parametric rate-oriented method 33 with associated point-wise confidence bounds. This method requires a constant number of examples of each label (low,not-low) in each fold, so we add examples to make N constant, and use random selection of examples to correct these frequencies, for instance by removing a randomly selected positive example and duplicating a randomly selected negative example in a particular fold. TPR: true positive rate; FPR: false positive rate.CC BY · PubMed Central
Figure 6. ROC curves for predicting article risk of bias. Generated with parametric rate-oriented method 33 with associated point-wise bounds. Blue dashed line: article contents (model 1); green solid line: PubMed title and abstract (model 3); TPR: true positive rate; FPR; false positive rate.
Figure 6. ROC curves for predicting article risk of bias. Generated with parametric rate-oriented method 33 with associated point-wise bounds. Blue dashed line: article contents (model 1); green solid line: PubMed title and abstract (model 3); TPR: true positive rate; FPR; false positive rate.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 36 references

Research works printed but not linked: 1

  1. Flach PA . ROC analysis. In: Encyclopedia of Machine Learning. New York, NY: Springer, 2011.

Cited by

See all 65

Metadata

Authors
Millard, L. A., Flach, P. A., & Higgins, J. P.
Affiliations
  • University of Bristol (GB)
  • MRC Integrative Epidemiology Unit (GB)
Corresponding author
Millard, Louise AC
Venue
International Journal of Epidemiology
Published
2015-12-08
Volume
45
Issue
1
Pages
266-277
ISSN
  • 0300-5771 (ISSN-L)
  • 1464-3685
OpenAlex
W2271587360
Cited by
65
References
36
Citations by year
2026: 5 · 2025: 10 · 2024: 3 · 2023: 11 · 2022: 5 · 2021: 9 · 2020: 5 · 2019: 6 · 2018: 4 · 2017: 5 · 2016: 2
FWCI
0.61
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • International Journal of Epidemiology (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Bristol Research (University of Bristol) (submittedVersion, cc-by)
  • Explore Bristol Research (submittedVersion, cc-by)
  • HighWire Press Open Archive (submittedVersion)
Funders
  • National Institute for Health and Care Research (NF-SI-0514-10114)
  • Medical Research Council (MR/K025643/1, MC_UU_12013, MC_UU_12013/1‐9, MCUU12013/1)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
  • Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
Keywords
risk of bias assessment, systematic reviews, text mining, sequence generation, allocation concealment, blinding, logistic regression
MeSH terms
Machine Learning, Humans, Review Literature as Topic, ROC Curve, Bias, Logistic Models, Data Mining, Datasets as Topic

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Machine learning to assist risk-of-bias assessments in systematic reviews. https://app.cochat.ai/papers/10.1093/ije/dyv306
The paper
Millard, L. A., Flach, P. A., & Higgins, J. P. (2016). Machine learning to assist risk-of-bias assessments in systematic reviews. International Journal of Epidemiology. https://doi.org/10.1093/ije/dyv306

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.