Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 36 of the 38 references it cites were found in a registry or an index.

Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews

Open access
Shemilt, Ian, Khan, Nada, Park, Sophie, Thomas, James · 2016 · Systematic Reviews

Abstract

BACKGROUND: Meta-research studies investigating methods, systems, and processes designed to improve the efficiency of systematic review workflows can contribute to building an evidence base that can help to increase value and reduce waste in research. This study demonstrates the use of an economic evaluation framework to compare the costs and effects of four variant approaches to identifying eligible studies for consideration in systematic reviews. METHODS: A cost-effectiveness analysis was conducted using a basic decision-analytic model, to compare the relative efficiency of 'safety first', 'double screening', 'single screening' and 'single screening with text mining' approaches in the title-abstract screening stage of a 'case study' systematic review about undergraduate medical education in UK general practice settings. Incremental cost-effectiveness ratios (ICERs) were calculated as the 'incremental cost per citation 'saved' from inappropriate exclusion' from the review. Resource use and effect parameters were estimated based on retrospective analysis of 'review process' meta-data curated alongside the 'case study' review, in conjunction with retrospective simulation studies to model the integrated use of text mining. Unit cost parameters were estimated based on the 'case study' review's project budget. A base case analysis was conducted, with deterministic sensitivity analyses to investigate the impact of variations in values of key parameters. RESULTS: Use of 'single screening with text mining' would have resulted in title-abstract screening workload reductions (base case analysis) of >60 % compared with other approaches. Across modelled scenarios, the 'safety first' approach was, consistently, equally effective and less costly than conventional 'double screening'. Compared with 'single screening with text mining', estimated ICERs for the two non-dominated approaches (base case analyses) ranged from £1975 ('single screening' without a 'provisionally included' code) to £4427 ('safety first' with a 'provisionally included' code) per citation 'saved'. Patterns of results were consistent between base case and sensitivity analyses. CONCLUSIONS: Alternatives to the conventional 'double screening' approach, integrating text mining, warrant further consideration as potentially more efficient approaches to identifying eligible studies for systematic reviews. Comparable economic evaluations conducted using other systematic review datasets are needed to determine the generalisability of these findings and to build an evidence base to inform guidance for review authors.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s13643-016-0315-4

Cites

36 of 36 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Four screening methods compared in the analysis
Fig. 1 Four screening methods compared in the analysisCC BY · PubMed Central
Fig. 2 Modelled flows of records and study reports through screening, with a ‘provisionally included’ code
Fig. 2 Modelled flows of records and study reports through screening, with a ‘provisionally included’ codeCC BY · PubMed Central
Fig. 3 Modelled flows of records and study reports through screening, without a ‘provisionally included’ code
Fig. 3 Modelled flows of records and study reports through screening, without a ‘provisionally included’ codeCC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

8 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 38 references

Research works printed but not linked: 3

  1. Higgins JPT, Deeks JJ, editors. Chapter 7: Selecting studies and collecting data. In: Higgins JPT, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions, Version 5.1.0 [updated March 2011]. http://handbook.cochrane.org . Accessed 1 July 2016.
  2. Cochrane methodology reviews. Cochrane Methodology Review Group. 2016. http://methodology.cochrane.org . Accessed 1 July 2016.
  3. Thomas J, Brunton J, Graziosi S. EPPI-Reviewer 4.0: software for research synthesis. London: EPPI-Centre Software, Social Science Research Unit, Institute of Education; 2010.

Cited by

See all 179

Metadata

Authors
Shemilt, I., Khan, N., Park, S., & Thomas, J.
Affiliations
University College London (GB)
Corresponding author
Shemilt, Ian
Venue
Systematic Reviews
Published
2016-08-17
Volume
5
Issue
1
Pages
140-140
ISSN
2046-4053 (ISSN-L)
OpenAlex
W2512040454
Cited by
168
References
38
Citations by year
2026: 11 · 2025: 17 · 2024: 24 · 2023: 22 · 2022: 17 · 2021: 27 · 2020: 9 · 2019: 19 · 2018: 15 · 2017: 7
FWCI
4.16
Type
journal article
Language
en
Open access
Locations
  • Systematic Reviews (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • UCL Discovery (University College London) (submittedVersion)
Funders
Medical Research Council (MR/J005037/1)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Health Systems, Economic Evaluations, Quality of Life
  • Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
Keywords
cost-effectiveness analysis, study identification, title and abstract screening, text mining, safety-first approach, incremental cost-effectiveness ratio, research waste, undergraduate medical education, general practice
MeSH terms
Cost-Benefit Analysis, Humans, Research Design, Data Mining, Patient Safety

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews. https://app.cochat.ai/papers/10.1186/s13643-016-0315-4
The paper
Shemilt, I., Khan, N., Park, S., & Thomas, J. (2016). Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews. Systematic Reviews. https://doi.org/10.1186/s13643-016-0315-4

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.