CoChat

Exploring issues in the conduct of website searching and other online sources for systematic reviews: how can we be systematic?

Open access
Stansfield, Claire, Dickson, Kelly, Bangpan, Mukdarut · 2016 · Systematic Reviews

The DOI matches its Crossref record; 33 of the 34 references it cites were found in a registry or an index; 7 of the 26 sources its claims cite were checked for support.

Abstract

Websites and online resources outside academic bibliographic databases can be significant sources for identifying literature, though there are challenges in searching and managing the results. These are pertinent to systematic reviews that are underpinned by principles of transparency, accountability and reproducibility. We consider how the conduct of searching these resources can be compatible with the principles of a systematic search. We present an approach to address some of the challenges. This is particularly relevant when websites are relied upon to identify important literature for a review. We recommend considering the process as three stages and having a considered rationale and sufficient recordkeeping at each stage that balances transparency with practicality of purpose. Advances in technology and recommendations for website providers are briefly discussed.

Overview

CoChat Summary

TL;DR

This discussion paper examines how website and online-resource searching can be conducted systematically within systematic reviews, where such sources often lack Boolean search, export, and search-history functions. It proposes treating website searching as three stages—planning, executing, and screening/information management—with considered rationale and proportionate recordkeeping to support transparency, accountability, and limited reproducibility.

Key findings

  • Across eight EPPI-Centre systematic reviews, “over a quarter of relevant citations were found from websites and internet search engines”; four reviews concerned international development interventions and four concerned people’s views for UK public health policy.
  • The authors propose three stages for website searching: (1) planning the search, (2) executing the search, and (3) screening records for relevance and managing results.
  • They recommend recording key elements such as resource name, searcher, date searched, pathway followed, search terms, whether predefined keywords were used, number of promising documents, number scanned, and screening approach.
  • They argue that reproducibility is limited for website searches because resource content, search functionality, and underlying search mechanisms are not static, and searching depends on user perspectives and skills; transparency and accountability should therefore be emphasized.

Methods

  • Discussion paper drawing on the authors’ experience conducting and supporting systematic reviews across health systems and social care, public health, education, social policy, and international development.
  • The paper presents a proposed approach and example recordkeeping tables, including an Excel worksheet example for searching the Alzheimer’s Society Research website and Rehabdata.
  • It reflects on related published guidance and case studies, including website searching, grey literature searching, Google Scholar, and trials registers.

Limitations

  • The paper states it does not aim to describe methods for using specific resources or to cover reporting the search in a published report, which it says is an area for separate consideration.
  • The paper states reproducibility is limited: the general approach may be replicated, but clear and systematic approaches “might not lead to replicable results,” and within each resource reproducibility is limited because content, functionality, and search mechanisms change and searching depends on user skills and perspectives.
  • Evident from the text: the proposed approach is based on the authors’ experience and examples rather than a formal empirical evaluation; the paper also notes that some review teams may find its recommended recordkeeping approach unacceptable, citing Rader et al. that some information specialists prefer to report every record precisely.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1186/s13643-016-0371-9

Cites

33 of 33 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

6 Valid · 2 Not valid · 11 Not checked · 42 Own results and interpretations

4 more claims

Create a free account to see all 61 claims and to verify the 19 references still open.

See all claims — free account
  • Probably not a substantial claim. 50
  • It may state two separate claims as one. 19

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 33 references

Research works printed but not linked: 3

  1. Lefebvre C, Manheimer E, Glanville J. Chapter 6: searching for studies. In: Higgins JPT, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 (updated March 2011). The Cochrane Collaboration, 2011. http://handbook.cochrane.org/ . Accessed 22 Mar 2016.
  2. Brunton G, Stansfield C, Caird J, Thomas J. Chapter: 6: finding relevant studies. In: Gough, D. Oliver S, Thomas J, editors. An introduction to systematic reviews. 2nd Edition. London: Sage Publications; 2017. In press.
  3. Collaboration for Environmental Evidence. Guidelines for systematic review and evidence synthesis in environmental management. Version 4.2. Environmental Evidence: Centre for Evidence-Based Conservation, Bangor University. 2013 www.environmentalevidence.org/Documents/Guidelines/Guidelines4.2.pdf , Accessed 22 Mar 2016.

Cited by

See all 124

Metadata

Authors
Stansfield, C., Dickson, K., & Bangpan, M.
Affiliations
University College London (GB)
Corresponding author
Stansfield, Claire
Venue
Systematic Reviews
Published
2016-11-15
Volume
5
Issue
1
Pages
191-191
ISSN
2046-4053 (ISSN-L)
OpenAlex
W2554603683
Cited by
122
References
34
Citations by year
2026: 16 · 2025: 18 · 2024: 13 · 2023: 14 · 2022: 13 · 2021: 17 · 2020: 8 · 2019: 7 · 2018: 10 · 2017: 6
Type
journal article
Language
en
Open access
Locations
  • Systematic Reviews (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • UCL Discovery (University College London) (submittedVersion)
Funders
University College London
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Social Sciences › Library and Information Sciences › Library Science and Information Literacy
Keywords
website searching, online resources, literature searching, transparency, reproducibility, record keeping, search strategies
MeSH terms
Humans, Research Design, Review Literature as Topic, Databases, Bibliographic, Information Storage and Retrieval, Publication Bias, Internet, Search Engine

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-05, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-05 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Exploring issues in the conduct of website searching and other online sources for systematic reviews: how can we be systematic? https://app.cochat.ai/papers/10.1186/s13643-016-0371-9
The paper
Stansfield, C., Dickson, K., & Bangpan, M. (2016). Exploring issues in the conduct of website searching and other online sources for systematic reviews: how can we be systematic? Systematic Reviews. https://doi.org/10.1186/s13643-016-0371-9

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.