Exploring issues in the conduct of website searching and other online sources for systematic reviews: how can we be systematic?
Open accessThe DOI matches its Crossref record; 33 of the 34 references it cites were found in a registry or an index; 7 of the 26 sources its claims cite were checked for support.
Abstract
Websites and online resources outside academic bibliographic databases can be significant sources for identifying literature, though there are challenges in searching and managing the results. These are pertinent to systematic reviews that are underpinned by principles of transparency, accountability and reproducibility. We consider how the conduct of searching these resources can be compatible with the principles of a systematic search. We present an approach to address some of the challenges. This is particularly relevant when websites are relied upon to identify important literature for a review. We recommend considering the process as three stages and having a considered rationale and sufficient recordkeeping at each stage that balances transparency with practicality of purpose. Advances in technology and recommendations for website providers are briefly discussed.
Overview
CoChat Summary
TL;DR
This discussion paper examines how website and online-resource searching can be conducted systematically within systematic reviews, where such sources often lack Boolean search, export, and search-history functions. It proposes treating website searching as three stages—planning, executing, and screening/information management—with considered rationale and proportionate recordkeeping to support transparency, accountability, and limited reproducibility.
Key findings
- Across eight EPPI-Centre systematic reviews, “over a quarter of relevant citations were found from websites and internet search engines”; four reviews concerned international development interventions and four concerned people’s views for UK public health policy.
- The authors propose three stages for website searching: (1) planning the search, (2) executing the search, and (3) screening records for relevance and managing results.
- They recommend recording key elements such as resource name, searcher, date searched, pathway followed, search terms, whether predefined keywords were used, number of promising documents, number scanned, and screening approach.
- They argue that reproducibility is limited for website searches because resource content, search functionality, and underlying search mechanisms are not static, and searching depends on user perspectives and skills; transparency and accountability should therefore be emphasized.
Methods
- Discussion paper drawing on the authors’ experience conducting and supporting systematic reviews across health systems and social care, public health, education, social policy, and international development.
- The paper presents a proposed approach and example recordkeeping tables, including an Excel worksheet example for searching the Alzheimer’s Society Research website and Rehabdata.
- It reflects on related published guidance and case studies, including website searching, grey literature searching, Google Scholar, and trials registers.
Limitations
- The paper states it does not aim to describe methods for using specific resources or to cover reporting the search in a published report, which it says is an area for separate consideration.
- The paper states reproducibility is limited: the general approach may be replicated, but clear and systematic approaches “might not lead to replicable results,” and within each resource reproducibility is limited because content, functionality, and search mechanisms change and searching depends on user skills and perspectives.
- Evident from the text: the proposed approach is based on the authors’ experience and examples rather than a formal empirical evaluation; the paper also notes that some review teams may find its recommended recordkeeping approach unacceptable, citing Rader et al. that some information specialists prefer to report every record precisely.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
6 Valid · 2 Not valid · 11 Not checked · 42 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“For some systematic reviews undertaken at the Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI-Centre), over a quarter of relevant citations were found from websites and internet search engines [4–13].”
§ Exploring issues in the conduct of website searching and other online sources for systematic reviews: how can we be systematic? > Background
What the sources say
- Not checked yetcites [4] Birdthistle I, Dickson K, Freeman M, Javidi L. What impact does the provision of separate toilets for girls at schools have on their primary and secondary school enrolment, attendance and completion? A systematic review of the evidence. London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London; 2011.
- Not checked yetcites [5] Brunton G, Oliver S, Oliver K, Lorenc T. A synthesis of research addressing children's, young people's and parents' views of walking and cycling for transport. London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London; 2006. http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=942 .
- Not checked yetcites [6] Brunton G, Wiggins M, Oakley A. Becoming a mother: a research synthesis of women's experiences of first-time motherhood. EPPI Centre, Social Science Research Unit, Institute of Education, University of London. 2011 http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=3295 .
- Not checked yetcites [7] Dickson K, Bangpan M. Providing access to economic assets for girls and young women in low-and-lower middle income countries: a systematic review of the evidence. London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London; 2012. http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=3359 . ISBN 978-1-907345-38-8.
- Partly supportedcites [8] Rees R, Oliver K, Woodman J, Thomas J. The views of young children in the UK about obesity, body size, shape and weight: a systematic review. BMC Public Health. 2011;11:188. sourceExperimental — accuracy not yet measured
The Methods > Search strategy section says this review searched sixteen websites alongside electronic databases and other sources. The Results section reports 11,128 citations screened and 28 studies included, but gives no breakdown by discovery source. It therefore does not support the claim that websites and internet search engines yielded over a quarter of relevant citations. (LLM, conf=0.78)
- Partly supportedcites [9] Rees R, Caird J, Dickson K, Vigurs C, Thomas J. ‘It’s on your conscience all the time’: a systematic review of qualitative studies examining views on obesity among young people aged 12–18 years in the UK. BMJ Open. 2014;4:e004404. doi: 10.1136/bmjopen-2013-004404 . sourceExperimental — accuracy not yet measured
The Methods section says this review searched 54 websites and used Internet search engines, so it supports the use of those sources in an EPPI-Centre review. Neither Methods nor Results reports what proportion of relevant citations those searches found; the claim that it exceeded a quarter is unsupported by our copy. (LLM, conf=0.84)
- Not checked yetcites [10] Stansfield C, Brunton G, Rees R. Search wide, dig deep: literature searching for qualitative research studies. An analysis of the publication formats and information sources used for four systematic reviews in public health. Res Synth Methods. 2014;5(2):142–51. source
- Not checked yetcites [11] Stansfield C, Dickson K. Locating evidence for developing countries: a case study of three public health reviews. Poster presentation at the 19th Cochrane Colloquium; 2011 Oct 19-22; Madrid, Spain [abstract] Cochrane Database of Syst Rev. 2011;Suppl(CD000003), 51-2, CMR-16617.
- Not checked yetcites [12] Stewart R, van Rooyen C, Dickson K, Majoro M, de Wet T. What is the impact of microfinance on poor people? A systematic review of evidence from sub-Saharan Africa. Technical report. London: EPPI-Centre, Social Science Research Unit, University of London; 2010.
- Not checked yetcites [13] Tripney J, Schucan Bird K, Kwan I, Kavanagh J. The impact of post-abortion care family planning counselling and services in low-income countries: a systematic review of the evidence. Technical report. London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London; 2011.
plainn = 4- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Every source the paper cites for this claim was checked and supports it.
In the paper
“We previously observed that relevant literature for low- and middle-income countries, such as working and policy papers, is often not included in databases, and is located from organisational websites, contacting authors or internet search engines [3].”
§ Exploring issues in the conduct of website searching and other online sources for systematic reviews: how can we be systematic? > Background
What the sources say
- Supportedcites [3] Oliver S, Bangpan M, Stansfield C, Stewart R. Capacity for conducting systematic reviews in low- and middle-income countries: a rapid appraisal. Health Res Policy Syst. 2015;13:23. doi: 10.1186/s12961-015-0012-0 . source
The Results section, under “Knowledge management systems,” says that much relevant literature, including working and policy papers, is absent from databases and is found through organisational websites, contacting authors, or internet search engines. The cited sentence closely paraphrases that passage in its LMIC-review context. (LLM, conf=0.95)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Glanville et al. [25] studied search approaches for two clinical trials registries, ICTRP and Clinicaltrials.gov, and found single-concept searches in the basic interfaces to be the most reliable.”
§ Exploring issues in the conduct of website searching and other online sources for systematic reviews: how can we be systematic? > Background > How can we be systematic?
What the sources say
- Supportedcites [25] Glanville JM, Duffy S, McCool R, Varley D. Searching ClinicalTrials.gov and the International Clinical Trials Registry Platform to inform systematic reviews: what are the optimal search approaches? J Med Libr Assoc. 2014;102(3):177–83. source
The Discussion says that highly sensitive single-concept searches in the basic interface seemed the most reliable approach in both resources for identifying known relevant records in the test reviews. The Search strategies section reports that this approach produced the best retrieval results. (LLM, conf=0.94)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Haddaway et al. [24] investigated approaches for searching Google Scholar, which only displays the first 1000 references of a search; they found that title searches enabled discovery of more grey literature (conference proceedings, theses, reports) than full-text searches.”
§ Exploring issues in the conduct of website searching and other online sources for systematic reviews: how can we be systematic? > Background > How can we be systematic?
What the sources say
- Supportedcites [24] Haddaway NR, Collins AM, Coughlin D, Kirk S. The role of Google Scholar in evidence reviews and its applicability to grey literature searching. PLoS One. 2015;10:e0138237. source
The Results report that title searches returned a higher proportion of grey literature than full-text searches, including more conference proceedings and theses. The Conclusions also identify reports among the grey literature found through title searches. The Other Considerations section states that only the first 1,000 Google Scholar results can be viewed. (LLM, conf=0.94)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“However, clear, systematic and replicable approaches to searching might not lead to replicable results, as observed by Adams et al. [30].”
§ Exploring issues in the conduct of website searching and other online sources for systematic reviews: how can we be systematic? > Background > Transparency, accountability and reproducibility
What the sources say
- Supportedcites [30] Adams J, Hillier-Brown F, Moore H, Lake A, Araujo-Soares V, White M, Summerbell C. Searching and synthesising ‘grey literature’ and ‘grey information’ in public health: critical reflections on three case studies. Syst Rev. 2016;5:164. source
The 'Replicability of searches' subsection (Results and discussion > Searching) states that although the team had clear plans describing comprehensive, systematic and replicable approaches to information searching, 'it is hard to claim that these led to replicable results', and that replication of their methods would be unlikely to yield the same results. The following summary line reinforces this: searching for grey literature and information 'can be systematic, but not necessarily replicable'. The paraphrase in the citing sentence matches the source, including its hedged framing ('might not'). (LLM, conf=0.93)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
4 more claims
Create a free account to see all 61 claims and to verify the 19 references still open.
- Probably not a substantial claim. 50
- It may state two separate claims as one. 19
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Searching general internet search engines and searching the contents of specific websites is a key component of many systematic reviews [26, 65].”
From PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews (2021) · cites this paper as [65]
The cited source says what the claim attributes to it.
In the citing paper
“Review teams may occasionally set an artificial limit to the number of items they will screen from a given search or source [65].”
From PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews (2021) · cites this paper as [65]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Exploring issues in the conduct of website searching and other online sources for systematic reviews: how can we be systematic? > Background > How can we be systematic?
“Godin et al. [15] describe an approach where they used a series of Google searches to identify 77 relevant organisations and websites.”
§ Exploring issues in the conduct of website searching and other online sources for systematic reviews: how can we be systematic? > References
“15. Collaboration for Environmental Evidence. Guidelines for systematic review and evidence synthesis in environmental management.”
Citations
Cites
The role of google scholar in evidence reviews and its applicability to grey literature searching
Haddaway, N. R., Collins, A. M., Coughlin, D., & Kirk, S. (2015) · PLOS ONE · doi:10.1371/journal.pone.0138237 · Cited by 2,387
Godin, K., Stapleton, J., Kirkpatrick, S. I., Hanning, R. M., & Leatherdale, S. T. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0125-0 · Cited by 959
Searching for grey literature for systematic reviews: challenges and benefits
Mahood, Q., Van Eerd, D., & Irvin, E. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1106 · Cited by 814
Adams, J., Hillier-Brown, F. C., Moore, H. J., Lake, A. A., Araujo-Soares, V., White, M., & Summerbell, C. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0337-y · Cited by 486
Hurley, M., Dickson, K., Hallett, R., Grant, R., Hauari, H., Walsh, N., Stansfield, C., & Oliver, S. (2018) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd010842.pub2 · Cited by 316
Self-citationNational Institute for Health and Clinical Excellence
(2008) · Nursing Standard · doi:10.7748/ns2008.08.22.50.30.p4587 · Cited by 260
Boeker, M., Vach, W., & Motschall, E. (2013) · BMC Medical Research Methodology · doi:10.1186/1471-2288-13-131 · Cited by 225
Bramer, W. M., Giustini, D., & Kramer, B. M. R. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0215-7 · Cited by 202
Shades of grey: Two forms of grey literature important for reviews in conservation
Haddaway, N. R., & Bayliss, H. R. (2015) · Biological Conservation · doi:10.1016/j.biocon.2015.08.018 · Cited by 180
Stewart, R. J., van Rooyen, C., Dickson, K., Majoro, M., & de Wet, T. (2010) · IOE EPrints · Cited by 178
Self-citation
Research works printed but not linked: 3
- Lefebvre C, Manheimer E, Glanville J. Chapter 6: searching for studies. In: Higgins JPT, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 (updated March 2011). The Cochrane Collaboration, 2011. http://handbook.cochrane.org/ . Accessed 22 Mar 2016.
- Brunton G, Stansfield C, Caird J, Thomas J. Chapter: 6: finding relevant studies. In: Gough, D. Oliver S, Thomas J, editors. An introduction to systematic reviews. 2nd Edition. London: Sage Publications; 2017. In press.
- Collaboration for Environmental Evidence. Guidelines for systematic review and evidence synthesis in environmental management. Version 4.2. Environmental Evidence: Centre for Evidence-Based Conservation, Bangor University. 2013 www.environmentalevidence.org/Documents/Guidelines/Guidelines4.2.pdf , Accessed 22 Mar 2016.
Cited by
Rethlefsen, M. L., Kirtley, S., Waffenschmidt, S., Ayala, A. P., Moher, D., Page, M. J., & Koffel, J. (2019) · doi:10.31219/osf.io/sfc38 · Cited by 3,696
Rethlefsen, M. L., Kirtley, S., Waffenschmidt, S., Ayala, A. P., Moher, D., Page, M. J., Koffel, J. B., PRISMA-S Group, Blunt, H., Brigham, T., et al. (2021) · Systematic Reviews · doi:10.1186/s13643-020-01542-z · Cited by 3,714
Meeting the review family: exploring review types and associated information retrieval requirements
Sutton, A., Clowes, M., Preston, L., & Booth, A. (2019) · Health Information & Libraries Journal · doi:10.1111/hir.12276 · Cited by 570
Rethlefsen, M. L., Kirtley, S., Waffenschmidt, S., Ayala, A. P., Moher, D., Page, M. J., Koffel, J. B., & Group, P.-S. (2021) · Journal of the Medical Library Association · doi:10.5195/jmla.2021.962 · Cited by 409
Mueller, M., D’Addario, M., Egger, M., Cevallos, M., Dekkers, O., Mugglin, C., & Scott, P. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-018-0495-9 · Cited by 400
Metadata
- Authors
- Stansfield, C., Dickson, K., & Bangpan, M.
- Affiliations
- University College London (GB)
- Corresponding author
- Stansfield, Claire
- Venue
- Systematic Reviews
- Published
- 2016-11-15
- Volume
- 5
- Issue
- 1
- Pages
- 191-191
- ISSN
- 2046-4053 (ISSN-L)
- PMC
- PMC5111285
- OpenAlex
- W2554603683
- Cited by
- 122
- References
- 34
- Citations by year
- 2026: 16 · 2025: 18 · 2024: 13 · 2023: 14 · 2022: 13 · 2021: 17 · 2020: 8 · 2019: 7 · 2018: 10 · 2017: 6
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Systematic Reviews (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- UCL Discovery (University College London) (submittedVersion)
- Funders
- University College London
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Social Sciences › Library and Information Sciences › Library Science and Information Literacy
- Keywords
- website searching, online resources, literature searching, transparency, reproducibility, record keeping, search strategies
- MeSH terms
- Humans, Research Design, Review Literature as Topic, Databases, Bibliographic, Information Storage and Retrieval, Publication Bias, Internet, Search Engine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-05, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-05 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Exploring issues in the conduct of website searching and other online sources for systematic reviews: how can we be systematic? https://app.cochat.ai/papers/10.1186/s13643-016-0371-9
- The paper
- Stansfield, C., Dickson, K., & Bangpan, M. (2016). Exploring issues in the conduct of website searching and other online sources for systematic reviews: how can we be systematic? Systematic Reviews. https://doi.org/10.1186/s13643-016-0371-9
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.