CoChat
Sign Up
Cited by 21Meta-analysis and systematic reviewsClinical practice guidelines implementationInformation Retrieval and Search Behavior

Machine learning to optimize literature screening in medical guideline development

Open access
Harmsen, Wouter, de Groot, Janke, Harkema, Albert, van Dusseldorp, Ingeborg, de Bruin, Jonathan, van den Brand, Sofie, van de Schoot, Rens·2024·Systematic Reviews
DOI 10.1186/s13643-024-02590-5PMC PMC11238391
Open this paper in CoChat — free account

Abstract

Objectives: In a time of exponential growth of new evidence supporting clinical decision-making, combined with a labor-intensive process of selecting this evidence, methods are needed to speed up current processes to keep medical guidelines up-to-date. This study evaluated the performance and feasibility of active learning to support the selection of relevant publications within medical guideline development and to study the role of noisy labels. Design: We used a mixed-methods design. Two independent clinicians’ manual process of literature selection was evaluated for 14 searches. This was followed by a series of simulations investigating the performance of random reading versus using screening prioritization based on active learning. We identified hard-to-find papers and checked the labels in a reflective dialogue. Main outcome measures: Inter-rater reliability was assessed using Cohen’s Kappa ( ĸ ). To evaluate the performance of active learning, we used the Work Saved over Sampling at 95% recall (WSS@95) and percentage Relevant Records Found at reading only 10% of the total number of records (RRF@10). We used the average time to discovery (ATD) to detect records with potentially noisy labels. Finally, the accuracy of labeling was discussed in a reflective dialogue with guideline developers. Results: Mean ĸ for manual title-abstract selection by clinicians was 0.50 and varied between − 0.01 and 0.87 based on 5.021 abstracts. WSS@95 ranged from 50.15% (SD = 17.7) based on selection by clinicians to 69.24% (SD = 11.5) based on the selection by research methodologist up to 75.76% (SD = 12.2) based on the final full-text inclusion. A similar pattern was seen for RRF@10, ranging from 48.31% (SD = 23.3) to 62.8% (SD = 21.20) and 65.58% (SD = 23.25). The performance of active learning deteriorates with higher noise. Compared with the final full-text selection, the selection made by clinicians or research methodologists deteriorated WSS@95 by 25.61% and 6.25%, respectively. Conclusion: While active machine learning tools can accelerate the process of literature screening within guideline development, they can only work as well as the input given by human raters. Noisy labels make noisy machine learning.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s13643-024-02590-5

Cites

40 of 40 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Active learning can be biased by the small number of initial screenings, leading to unrepresentative prioritizations.

    “However, active learning has inherent limitations: it can be biased by the small number of initial screenings, leading to unrepresentative prioritizations 21 .”

    From AIM review tool: artificial intelligence for smarter systematic review screening (2026) · cites this paper as [21]

    The Methods > Simulation passage says the authors varied the initial relevant paper across runs to mitigate potential bias from the starting paper in the model’s first iteration. This supports sensitivity to initial training records, but the paper does not say that a small number of initial screenings causes unrepresentative prioritizations. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Harmsen, W., de Groot, J., Harkema, A., van Dusseldorp, I., de Bruin, J., van den Brand, S., & van de Schoot, R.
Affiliations
  • Federatie Medisch Specialisten (NL)
  • Utrecht University (NL)
ORCID
Venue
Systematic Reviews
Publisher
Springer Science and Business Media LLC
Published
2024-07-11
Volume
13
Issue
1
Pages
177-177
ISSN
2046-4053 (ISSN-L)
DOI
10.1186/s13643-024-02590-5
PMC
PMC11238391
OpenAlex
W4400557303
Cited by
21
References
40
Citations by year
2026: 11 · 2025: 10
FWCI
5.83
Type
journal article
Language
en
Open access
Licence
Locations
  • Systematic Reviews (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
  • Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
Funders
  • Nederlandse Organisatie voor Wetenschappelijk Onderzoek (2020/SGW/00909334)
  • ZonMw (project 516022528)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
Keywords
active learning (machine learning), literature screening, label noise, evidence selection, title and abstract screening, inter-rater reliability, reflective dialogue
MeSH terms
Clinical Decision-Making, Machine Learning, Humans, Reproducibility of Results, Practice Guidelines as Topic, Evidence-Based Medicine

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review