Searching for Programme theories for a realist evaluation: a case study comparing an academic database search and a simple Google search
Open accessPublished version of a preprint
Abstract
Background: Realist methodologies are increasingly being used to evaluate complex interventions in health and social care. Programme theory (ideas and assumptions of how a particular intervention works) development is the first step in a realist evaluation or a realist synthesis, with literature reviews providing important evidence to support this. Deciding how to search for programme theories is challenging and there is limited guidance available. Using an example of identifying programme theories for a realist evaluation of Pressure Ulcer Risk Assessment Instruments in clinical practice, the authors explore and compare several different approaches to literature searching and highlight important methodological considerations for those embarking on a programme theory review. Methods: We compared the performance of an academic database search with a simple Google search and developed an optimised search strategy for the identification primary references (i.e. documents providing the clearest examples of programme theories) associated with the use of Pressure Ulcer Risk Assessment Instruments (PU-RAIs). We identified the number of primary references and the total number of references retrieved per source. We then calculated the number needed to read (NNR) expressed as the total number of titles and abstracts screened to identify one relevant reference from each source. Results: The academic database search (comprising CINAHL, The Cochrane Library, EMBASE, HMIC, Medline) identified 2 /10 primary references with a NNR of 1395.The Google search identified 7/10 primary references with a NNR of 10.1. The combined NNR was 286.3. The optimised search combining Google and CINAHL identified 10/10 primary references with a NNR of 40.2. Conclusion: The striking difference between the efficiency of the review’s academic database and Google searches in finding relevant references prompted an in-depth comparison of the two types of search. The findings indicate the importance of including grey literature sources such as Google in this particular programme theory search, while acknowledging the need for transparency of methods. Further research is needed to facilitate improved guidance for programme theory searches to enhance practice in the realist field and to save researcher time and therefore resource.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Process evaluation of complex interventions: Medical Research Council guidance
Moore, G. F., Audrey, S., Barker, M., Bond, L., Bonell, C., Hardeman, W., Moore, L., O'Cathain, A., Tinati, T., Wight, D., et al. (2015) · BMJ · doi:10.1136/bmj.h1258 · Cited by 6,204 · see citing works
The role of google scholar in evidence reviews and its applicability to grey literature searching
Haddaway, N. R., Collins, A. M., Coughlin, D., & Kirk, S. (2015) · PLOS ONE · doi:10.1371/journal.pone.0138237 · Cited by 2,387 · see citing works
Greenhalgh, T., & Peacock, R. (2005) · BMJ · doi:10.1136/bmj.38636.593461.68 · Cited by 2,045 · see citing works
Systematic Approaches to a Successful Literature Review
Booth, A., Sutton, A., Clowes, M., & James, M. M.-S. (2022) · SAGE Publications Ltd eBooks · doi:10.4135/9781036232009 · Cited by 1,174 · see citing works
RAMESES II reporting standards for realist evaluations
Wong, G., Westhorp, G., Manzano, A., Greenhalgh, J., Jagosh, J., & Greenhalgh, T. (2016) · BMC Medicine · doi:10.1186/s12916-016-0643-1 · Cited by 880 · see citing works
Self-citationJournal of Multidisciplinary Healthcare
(2020) · doi:10.2147/jmdh · Cited by 644 · see citing works
Adams, J., Hillier-Brown, F. C., Moore, H. J., Lake, A. A., Araujo-Soares, V., White, M., & Summerbell, C. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0337-y · Cited by 486 · see citing works
Complex Interventions in Health
Richards, D. A. (2015) · Routledge eBooks · doi:10.4324/9780203794982 · Cited by 332 · see citing works
Papaioannou, D., Sutton, A., Carroll, C., Booth, A., & Wong, R. (2010) · Health Information & Libraries Journal · doi:10.1111/j.1471-1842.2009.00863.x · Cited by 318 · see citing works
Risk assessment tools for the prevention of pressure ulcers
Moore, Z. E. H., & Cowman, S. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd006471.pub3 · Cited by 238 · see citing works
Greenhalgh, J., Dalkin, S. M., Gooding, K., Gibbons, E. J., Wright, J. M., Meads, D. M., Black, N. A., Valderas, J. M., & Pawson, R. (2017) · Health Services and Delivery Research · doi:10.3310/hsdr05020 · Cited by 238 · see citing works
Self-citationSystematic searching for theory to inform systematic reviews: is it feasible? Is it desirable?
Booth, A., & Carroll, C. (2015) · Health Information & Libraries Journal · doi:10.1111/hir.12108 · Cited by 196 · see citing works
Protocol—the RAMESES II study: developing guidance and reporting standards for realist evaluation
Greenhalgh, T., Wong, G., Jagosh, J., Greenhalgh, J., Manzano, A., Westhorp, G., & Pawson, R. (2015) · BMJ Open · doi:10.1136/bmjopen-2015-008567 · Cited by 193 · see citing works
Self-citationSearching multiple databases for systematic reviews: added value or diminishing returns?
Stevinson, C., & Lawlor, D. A. (2004) · Complementary Therapies in Medicine · doi:10.1016/j.ctim.2004.09.003 · Cited by 143 · see citing works
Stansfield, C., Dickson, K., & Bangpan, M. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0371-9 · Cited by 122 · see citing works
Systematic reviews of test accuracy should search a range of databases to identify primary studies
Whiting, P. F., Westwood, M. E., Burke, M., Sterne, J. A. C., & Glanville, J. (2007) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.05.013 · Cited by 112 · see citing works
Reviewing and selecting outcome measures for use in routine practice
Greenhalgh, J., Long, A., Brettle, A., & Grant, M. J. (1998) · Journal of Evaluation in Clinical Practice · doi:10.1111/j.1365-2753.1998.tb00097.x · Cited by 91 · see citing works
Self-citationScoping and Searching to Support Realist Approaches
Booth, A., Wright, J. M., & Briscoe, S. (2018) · SAGE Publications Ltd eBooks · doi:10.4135/9781526451729.n10 · Cited by 82 · see citing works
Self-citationCooper, C., Lovell, R., Husk, K., Booth, A., & Garside, R. (2017) · Research Synthesis Methods · doi:10.1002/jrsm.1286 · Cited by 66 · see citing works
Do pressure ulcer risk assessment scales improve clinical practice?
Kottner, J. (2010) · Journal of Multidisciplinary Healthcare · doi:10.2147/jmdh.s9286 · Cited by 64 · see citing works
Ćurković, M., & Košec, A. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-018-0599-2 · Cited by 62 · see citing works
Randell, R., Greenhalgh, J., Hindmarsh, J., Dowding, D., Jayne, D. G., Pearman, A. D., Gardner, P. H., Croft, J., & Kotzé, A. (2014) · Implementation Science · doi:10.1186/1748-5908-9-52 · Cited by 60 · see citing works
Self-citationBriscoe, S. (2015) · BMC Research Notes · doi:10.1186/s13104-015-1079-y · Cited by 57 · see citing works
Levay, P., Ainsworth, N., Kettle, R., & Morgan, A. (2016) · Research Synthesis Methods · doi:10.1002/jrsm.1158 · Cited by 50 · see citing works
The Today Programme’s Contribution to Evidence-based Policy
Pawson, R., Owen, L., & Wong, G. (2010) · Evaluation · doi:10.1177/1356389010369636 · Cited by 22 · see citing works
Wright, J. M., Cottrell, D. J., & Mir, G. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.02.017 · Cited by 16 · see citing works
Self-citationJohansen, E. A., Moore, Z. E. H., van Etten, M. T., & Strapp, H. (2014) · Journal of Wound Care · doi:10.12968/jowc.2014.23.7.369 · Cited by 16 · see citing works
Pressure ulcer risk assessment.
Guy, H. (2012) · PubMed · Cited by 13 · see citing works
Booth, A. (2013) · White Rose eTheses Online (University of Leeds, The University of Sheffield, University of York) · Cited by 12 · see citing works
Complex interventions in health an overview of research methods
Richards, D. A., & Hallberg, I. R. (2015) · Cited by 12 · see citing works
Risk Assessment Scales for Predicting the Risk of Developing Pressure Ulcers
Torra‐Bou, J. E., García‐Fernández, F. P., Pancorbo-Hidalgo, P. L., & Furtado, K. (2006) · Springer eBooks · doi:10.1007/1-84628-134-2_6 · Cited by 10 · see citing works
(2018) · Facet eBooks · doi:10.29085/9781783303755 · Cited by 9 · see citing works
Pressure ulcer risk assessment and grading.
Guy, H. (2007) · PubMed · Cited by 9 · see citing works
Green, H. A. (2017) · Lincoln Repository (University of Lincoln) · Cited by 2 · see citing works
Flagged first, then by citations
1 of the 35 references couldn’t be matched to a known paper.
Research works printed but not linked: 6
- Pawson R, Tilley N. Realist Evaluation. Editor: B.C. Office; 2004. http://www.communitymatters.com.au/RE_chapter.pdf.
- Lefebvre C, et al., Chapter 6: Searching for studies. In: Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available from www.cochrane-handbook.org, 2011.
- Moore ZE, Cowman S. Risk assessment tools for the prevention of pressure ulcers. Cochrane Database Syst Rev. 2014;2:CD006471.
- Bethel A and R. M, Search summary table for systematic reviews (SRs). In: Filtering the information overload for better decisions. , in Abstracts of the 23rd Cochrane Colloquium. 2015, John Wiley & Sons: Vienna, Austria. .
- Fletcher J. An overview of pressure ulcer risk assessment tools. Wounds UK. 2017;13(1):18–26.
- NPUAP/EPUAP/PPPIA, Risk factors and risk assessment chapter..
Cited by
Sorted by citations
Searching for studies: A guide to information retrieval for Campbell systematic reviews
MacDonald, H. R., Comer, C., Foster, M., Labelle, P., Marsalis, S. E., Nyhan, K., Premji, Z., Rogers, M., Splenda, R., Stansfield, C., et al. (2024) · Campbell Systematic Reviews · doi:10.1002/cl2.1433 · Cited by 105 · see citing works
Heath, A., Levay, P., & Tuvey, D. (2021) · Health Information & Libraries Journal · doi:10.1111/hir.12414 · Cited by 49 · see citing works
Briscoe, S., Abbott, R., Lawal, H., Shaw, L., & Coon, J. T. (2023) · Research Synthesis Methods · doi:10.1002/jrsm.1622 · Cited by 41 · see citing works
Intelligent Process Automation and Business Continuity: Areas for Future Research
Brás, J. C., Pereira, R., & Moro, S. (2023) · Information · doi:10.3390/info14020122 · Cited by 26 · see citing works
Identifying evidence for five realist reviews in primary health care: A comparison of search methods
Duddy, C., & Roberts, N. W. (2021) · Research Synthesis Methods · doi:10.1002/jrsm.1523 · Cited by 23 · see citing works
Advances in auditing and business continuity: A study in financial companies
Brás, J. C., Pereira, R., Fonseca, M., Ribeiro, R., & Bianchi, I. S. (2024) · Journal of Open Innovation Technology Market and Complexity · doi:10.1016/j.joitmc.2024.100304 · Cited by 18 · see citing works
Supplementary databases increased literature search coverage beyond PubMed and Embase
Frandsen, T. F., Moos, C. M., Marino, C. I. L. H., & Eriksen, M. B. (2025) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2025.111704 · Cited by 16 · see citing works
Balancing Business, IT, and Human Capital: RPA Integration and Governance Dynamics
Brás, J. C., Pereira, R., Melo, M., Bianchi, I. S., & Ribeiro, R. (2025) · Information · doi:10.3390/info16090793 · Cited by 5 · see citing works
Integrated mental health for refugees: A realist theory building study
Clark, N., Bullon, A. A., Huq, M., & Mukumbang, F. C. (2026) · PLOS mental health. · doi:10.1371/journal.pmen.0000547 · Cited by 5 · see citing works
Lukac, S., Griewing, S., Leinert, E., Dayan, D., Heitmeir, B., Wallwiener, M., Janni, W., Fink, V., & Ebner, F. K. (2024) · Clinics and Practice · doi:10.3390/clinpract15010008 · Cited by 5 · see citing works
Barnard, M., Casey, M., & O’Connor, L. (2021) · HRB Open Research · doi:10.12688/hrbopenres.13451.1 · Cited by 3 · see citing works
Improving communication of prognostic uncertainty in advanced illness: a realist review
Etkind, S. N., Weetman, K., Hyams, K., Acharya, T., Shokraneh, F., & MacArtney, J. I. (2026) · BMJ Open · doi:10.1136/bmjopen-2025-115385 · Cited by 3 · see citing works
How realist reviews might be helpful to further insights in Problem-Based Learning
Bendermacher, G., Dolmans, D. H. J. M., & Egbrink, M. G. A. O. (2023) · Interdisciplinary Journal of Problem-based Learning · doi:10.14434/ijpbl.v17i2.35636 · Cited by 3 · see citing works
Using PURPOSE-T in clinical practice: A realist evaluation
Coleman, S. B., Greenhalgh, J., Schoonhoven, L., Twiddy, M., & Nixon, J. (2024) · Journal of Tissue Viability · doi:10.1016/j.jtv.2024.06.014 · Cited by 3 · see citing works
Self-citationFearrien, B. D. (2022) · Journal of Religious & Theological Information · doi:10.1080/10477845.2022.2047381 · Cited by 2 · see citing works
Hunter, R. L., Booth, A., & Wood, L. (2026) · BMC Medical Research Methodology · doi:10.1186/s12874-026-02814-3 · Cited by 2 · see citing works
Renshaw, J., Halter, M., & Quinn, T. (2022) · British Paramedic Journal · doi:10.29045/14784726.2022.03.6.4.48 · Cited by 1 · see citing works
Jagosh, J., Briscoe, S., Greenley, S. L., Edwards, A., Klok, F. A., Johnson, M., Noble, S., & Pearson, M. (2026) · Research Synthesis Methods · doi:10.1017/rsm.2025.10068
Clark, N., Bullon, A. A., Huq, M., & Mukumbang, F. C. (2026) · BMJ Open · doi:10.1136/bmjopen-2025-111367
Bullon, A. A., Lobban, F. A., & Harding, A. J. E. (2026) · Research Involvement and Engagement · doi:10.1186/s40900-026-00980-8
- Authors
- Coleman, S., Wright, J. M., Nixon, J., Schoonhoven, L., Twiddy, M., & Greenhalgh, J.
- Affiliations
- University of Leeds (GB)
- University Medical Center Utrecht (NL)
- University of Hull (GB)
- Corresponding author
- Coleman, Susanne
- ORCID
- Venue
- BMC Medical Research Methodology
- Published
- 2020-08-26
- Volume
- 20
- Issue
- 1
- Pages
- 217-217
- ISSN
- 1471-2288 (ISSN-L)
- PMC
- PMC7450563
- OpenAlex
- W3044445168
- Cited by
- 20
- References
- 35
- Citations by year
- 2026: 6 · 2025: 2 · 2024: 4 · 2023: 3 · 2022: 2 · 2021: 3
- FWCI
- 1.21
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- BMC Medical Research Methodology (publishedVersion, cc-by)
- Research Square (acceptedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Repository@Hull (Worktribe) (University of Hull) (publishedVersion, other-oa)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
- Repository@Hull (Worktribe) (University of Hull) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Research Trainees Coordinating Centre (PDF 2016-09-054)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Decision Sciences › Management Science and Operations Research › Evaluation and Performance Assessment
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Keywords
- program theory, realist evaluation, literature searching, Google search, number needed to read, grey literature, complex interventions
- MeSH terms
- Humans, Publications, Databases, Factual, MEDLINE
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
